Oct 30 19:03:25.243590 ip-10-0-130-164 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 19:03:25.243600 ip-10-0-130-164 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 19:03:25.243607 ip-10-0-130-164 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 19:03:25.243825 ip-10-0-130-164 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 19:03:35.264838 ip-10-0-130-164 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 19:03:35.264852 ip-10-0-130-164 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d7498e4ed21b45d59f11e046e22f5945 --
Oct 30 19:05:22.936542 ip-10-0-130-164 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:05:23.360315 ip-10-0-130-164 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:05:23.360315 ip-10-0-130-164 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:05:23.360315 ip-10-0-130-164 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:05:23.360315 ip-10-0-130-164 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:05:23.363785 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363694    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:23.363785 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363784    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:23.363785 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363788    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363794    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363799    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363803    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363806    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363808    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363811    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363814    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363816    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363819    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363822    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363824    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363826    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363829    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363832    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363834    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363836    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363847    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363850    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:23.363877 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363852    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363855    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363858    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363860    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363863    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363866    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363868    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363871    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363873    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363876    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363878    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363881    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363884    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363886    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363889    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363892    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363894    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363897    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363899    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363901    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:23.364336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363904    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363906    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363909    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363911    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363913    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363916    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363919    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363921    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363923    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363928    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363931    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363934    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363938    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363941    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363943    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363946    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363948    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363951    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363953    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363956    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:23.364796 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363958    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363961    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363963    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363966    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363968    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363970    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363973    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363975    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363978    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363980    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363983    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363985    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363988    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363990    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363993    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363995    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.363997    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364000    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364002    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364004    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:23.365300 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364007    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364010    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364012    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364014    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364017    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364020    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364024    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364027    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364030    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364032    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364034    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364037    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364039    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364042    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364044    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364047    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364049    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364051    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364054    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:23.365890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364056    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364059    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364061    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364063    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364067    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364069    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364892    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364900    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364903    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364906    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364909    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364912    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364915    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364918    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364920    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364923    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364926    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364929    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364933    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364936    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:23.366603 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364939    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364941    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364944    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364946    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364949    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364952    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364955    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364957    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364960    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364962    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364965    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364967    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364970    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364972    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364975    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364977    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364980    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364983    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364985    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364988    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:23.367492 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364990    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364993    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.364996    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365000    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365003    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365006    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365009    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365012    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365014    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365017    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365019    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365023    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365025    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365028    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365030    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365033    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365035    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365038    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365040    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:23.368394 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365043    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365046    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365048    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365051    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365055    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365058    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365060    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365062    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365065    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365069    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365072    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365075    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365078    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365080    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365083    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365086    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365088    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365091    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365093    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:23.369191 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365096    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365099    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365101    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365104    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365106    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365109    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365112    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365114    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365117    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365119    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365122    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365124    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365127    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365129    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365132    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365134    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365137    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365140    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365142    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:23.369939 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365145    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365147    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365150    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365152    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365154    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365157    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365159    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365162    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365165    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365167    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365171    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365174    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365176    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365178    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.365181    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366777    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366790    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366796    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366801    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:23.370499 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366805    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366810    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366814    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366818    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366822    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366826    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366831    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366835    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366839    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366844    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366848    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366853    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366857    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366861    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366865    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366869    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366873    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366878    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366882    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366887    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:23.371150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366891    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366896    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366900    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366904    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366908    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366918    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366923    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366927    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366931    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366936    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366941    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366945    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366949    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366953    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366957    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366961    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366965    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366970    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366974    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366978    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:23.371687 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366983    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366987    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366991    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366995    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.366999    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367003    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367007    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367012    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367016    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367020    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367026    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367032    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367039    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367045    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367051    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367056    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367061    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367065    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367069    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:23.372164 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367073    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367077    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367082    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367087    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367092    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367096    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367100    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367104    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367108    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367115    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367122    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367127    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367132    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367136    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367140    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367144    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367149    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367154    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367158    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:23.372718 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367163    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367168    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367173    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367177    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367181    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367186    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367190    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367194    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367199    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367203    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367207    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367211    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367215    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367220    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367224    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367229    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367234    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367238    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367242    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367246    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:23.373150 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367250    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367254    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367258    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.367263    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367419    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367431    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367441    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367448    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367455    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367461    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367467    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367474    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367479    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367484    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367490    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367495    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367500    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367504    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367509    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367513    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367517    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367522    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367529    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367534    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:05:23.373643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367539    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367544    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367549    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367556    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367561    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367566    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367571    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367576    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367581    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367586    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367591    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367595    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367602    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367607    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367612    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367617    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367622    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367627    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367633    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367639    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367643    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367648    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367653    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367661    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:05:23.374191 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367666    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367670    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367675    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367680    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367684    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367689    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367693    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367698    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367702    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367707    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367714    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367719    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367724    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367729    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367734    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367739    2577 flags.go:64] FLAG: --help="false"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367744    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367750    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367754    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367759    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367764    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367777    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367782    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:05:23.374943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367787    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367792    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367797    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367802    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367807    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367812    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367817    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367821    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367826    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367831    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367836    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367841    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367845    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367850    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367859    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367864    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367869    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367873    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367879    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367885    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367890    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367894    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367901    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367906    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367912    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:05:23.375587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367917    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367921    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367925    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367930    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367935    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367939    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367944    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367956    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367960    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367965    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367970    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367975    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367979    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367984    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367988    2577 flags.go:64] FLAG: --port="10250"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367993    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.367998    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02c0239956684b658"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368004    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368008    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368013    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368018    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368024    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368028    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368033    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368037    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368046    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:05:23.376173 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368051    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368059    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368064    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368068    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368073    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368078    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368083    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368088    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368093    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368098    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368103    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368108    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368112    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368117    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368121    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368125    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368130    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368135    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368140    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368150    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368155    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368160    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368167    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368171    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368177    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:05:23.376809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368181    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368186    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368191    2577 flags.go:64] FLAG: --v="2"
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368198    2577 flags.go:64] FLAG: --version="false"
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368204    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368210    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.368217    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368498    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368510    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368516    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368520    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368525    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368530    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368534    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368538    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368542    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368546    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368550    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368555    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368559    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368564    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:23.377402 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368568    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368572    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368576    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368580    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368584    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368589    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368593    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368597    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368602    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368606    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368610    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368616    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368620    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368624    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368628    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368632    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368636    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368642    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368646    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368652    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:23.377890 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368656    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368660    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368665    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368669    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368673    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368677    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368689    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368693    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368697    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368701    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368705    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368710    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368714    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368719    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368723    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368727    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368731    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368735    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368739    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368743    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:23.378375 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368747    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368752    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368756    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368760    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368768    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368775    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368781    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368786    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368791    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368797    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368801    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368807    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368811    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368816    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368821    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368825    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368829    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368833    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:23.378847 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368838    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368842    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368846    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368850    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368855    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368859    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368863    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368868    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368872    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368876    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368881    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368885    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368889    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368893    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368898    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368902    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368906    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368910    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368914    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368919    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:23.379378 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368924    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368928    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368932    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368938    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368942    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368948    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368952    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368956    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368960    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368965    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368969    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368973    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368977    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.368981    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.369022    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:05:23.379842 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369086    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369095    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369101    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369105    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369109    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369114    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369118    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369123    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369127    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369131    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369136    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369140    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369145    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369149    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369153    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369157    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369161    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369166    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369170    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369175    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:23.380192 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369178    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369185    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369191    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369195    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369199    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369203    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369207    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369211    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369215    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369219    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369223    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369228    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369232    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369236    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369240    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369245    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369249    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369254    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369258    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:23.380679 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369262    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369266    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369270    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369293    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369297    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369301    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369305    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369309    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369313    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369318    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369322    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369327    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369331    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369335    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369341    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369347    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369351    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369355    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369360    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369363    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:23.381114 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369368    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369372    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369377    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369381    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369385    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369389    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369393    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369397    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369401    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369405    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369410    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369416    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369421    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369425    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369430    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369434    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369439    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369443    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369447    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369451    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:23.381599 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369455    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369459    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369464    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369468    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369473    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369477    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369483    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369489    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369493    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369497    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369501    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369505    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369510    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369514    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369518    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369524    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369530    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369537    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:23.382050 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369542    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369547    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369552    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369556    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369560    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369564    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369568    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369575    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.369579    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.377070    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.377085    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.377101    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.377106    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377173    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377179    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377183    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377186    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377188    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377191    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:23.382493 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377194    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377196    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377199    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377202    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377204    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377207    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377210    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377213    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377215    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377218    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377220    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377223    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377225    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377228    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377230    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377233    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377235    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377238    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377240    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:23.382935 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377242    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377245    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377247    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377250    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377256    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377259    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377263    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377266    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377268    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377271    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377296    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377299    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377302    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377304    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377307    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377309    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377312    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377314    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377317    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:23.383393 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377319    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377321    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377324    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377326    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377329    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377331    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377334    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377336    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377339    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377342    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377344    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377347    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377350    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377353    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377355    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377357    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377360    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377363    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377366    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377370    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:23.383841 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377373    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377376    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377379    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377382    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377385    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377387    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377390    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377392    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377395    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377397    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377400    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377402    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377405    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377407    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377410    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377412    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377415    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377418    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377420    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:23.384359 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377423    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377426    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377429    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377431    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377433    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377436    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377438    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377441    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377443    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377446    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377449    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377452    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377455    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377457    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377460    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377462    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377464    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377467    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377469    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:23.384813 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377471    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377474    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377476    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377478    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.377506    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377691    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377696    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377700    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377702    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377705    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377708    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377710    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377712    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377715    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377717    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377720    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:23.385254 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377722    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377725    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377728    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377730    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377733    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377735    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377738    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377741    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377743    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377745    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377749    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377751    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377754    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377756    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377759    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377761    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377764    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377766    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377769    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377771    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:23.385658 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377774    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377776    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377779    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377781    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377784    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377786    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377789    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377791    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377794    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377796    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377799    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377802    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377804    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377807    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377809    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377812    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377814    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377817    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377820    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:23.386121 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377822    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377825    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377827    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377829    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377837    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377840    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377843    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377845    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377849    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377853    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377855    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377858    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377861    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377863    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377866    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377869    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377871    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377873    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377876    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:23.386612 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377878    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377882    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377886    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377889    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377892    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377895    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377898    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377901    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377904    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377906    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377909    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377911    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377913    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377916    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377918    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377921    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377923    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377926    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377929    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:23.387054 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377931    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377934    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377937    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377939    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377942    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377944    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377947    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377949    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377951    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377954    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377956    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377959    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377961    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377964    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377966    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377969    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377971    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:23.387513 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.377974    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.377997    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378031    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378034    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378037    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378040    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378042    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378044    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378047    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378049    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378052    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378054    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378057    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378059    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378062    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378065    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:05:23.387914 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378068    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378071    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378073    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378075    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378078    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378080    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378082    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378085    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378087    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378090    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378092    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378095    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378097    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378099    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378102    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378104    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378107    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378110    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378112    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378115    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:05:23.388294 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378117    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378120    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378122    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378125    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378127    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378129    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378132    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378134    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378136    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378139    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378141    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378144    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378154    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378157    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378160    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378163    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378165    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378168    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378170    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378173    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:05:23.388748 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378175    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378178    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378180    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378182    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378186    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378189    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378192    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378194    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378197    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378200    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378202    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378205    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378207    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378210    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378212    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378214    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378218    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378221    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378224    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:05:23.389229 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378227    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378229    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378232    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378234    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378237    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378239    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378243    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378245    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378248    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378250    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378252    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378255    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378257    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378259    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378262    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378264    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378267    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378269    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378271    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:05:23.389693 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378288    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378291    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378295    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378298    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378300    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378303    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378305    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378308    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378310    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378313    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378316    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378319    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378321    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:23.378324    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.378380    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.380897    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.381919    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.382019    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:05:23.390131 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.382052    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:05:23.390577 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.384639    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:05:23.390577 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.387783    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:05:23.401572 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.401555    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:05:23.405393 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.405377    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 19:05:23.406230 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.406205    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:05:23.410537 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.410516    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7c290a57-105a-4130-a477-f6225d60ac2f:/dev/nvme0n1p4 e99e1900-b542-4f09-aa3c-50b6724e5909:/dev/nvme0n1p3]
Oct 30 19:05:23.410591 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.410539    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:05:23.417168 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.417061    2577 manager.go:217] Machine: {Timestamp:2026-10-30 19:05:23.415743739 +0000 UTC m=+0.370875159 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100438 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e7d76f9dae2eaf75579ff7893a50d SystemUUID:ec2e7d76-f9da-e2ea-f755-79ff7893a50d BootID:d7498e4e-d21b-45d5-9f11-e046e22f5945 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:f8:1b:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:f8:1b:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:2e:f3:c9:99:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:05:23.417168 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.417166    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:05:23.417288 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.417233    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:05:23.418011 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.417994    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:05:23.419165 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.419139    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:05:23.419310 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.419169    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-164.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:05:23.419353 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.419313    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:05:23.419353 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.419321    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:05:23.419412 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.419354    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:05:23.419412 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.419372    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:05:23.420500 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.420489    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:05:23.424644 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.424632    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:05:23.424767 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.424758    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:05:23.424796 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.424790    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:05:23.424825 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.424806    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:05:23.424825 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.424819    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:05:23.424886 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.424828    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:05:23.424886 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.424839    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:05:23.425178 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.425167    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:05:23.427488 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.427473    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:05:23.433986 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.433971    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:05:23.434886 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.434874    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:05:23.434919 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.434904    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:05:23.435051 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435041    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:05:23.435080 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435057    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:05:23.435080 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435063    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:05:23.435080 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435068    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:05:23.435080 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435074    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:05:23.435080 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435080    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:05:23.435815 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435806    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:05:23.435845 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435816    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:05:23.435845 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435823    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:05:23.435845 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435828    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:05:23.435845 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435842    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:05:23.436051 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.435851    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:05:23.436567 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.436558    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:05:23.436596 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.436568    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:05:23.439755 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.439728    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-164.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:05:23.440594 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.440581    2577 server.go:1263] "Started kubelet"
Oct 30 19:05:23.440673 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.440648    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:05:23.440813 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.440761    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:05:23.440888 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.440840    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:05:23.441357 ip-10-0-130-164 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:05:23.442411 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.442351    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:05:23.443012 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.443000    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:05:23.449334 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.449311    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:05:23.455814 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.455792    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-164.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:05:23.455895 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.455803    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 19:05:23.457158 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.455721    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-164.us-east-2.compute.internal.18e36416456f2c31  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-164.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-164.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:05:23.440561201 +0000 UTC m=+0.395692624,LastTimestamp:2026-10-30 19:05:23.440561201 +0000 UTC m=+0.395692624,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-164.us-east-2.compute.internal,}"
Oct 30 19:05:23.457933 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.457914    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:05:23.458013 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.457939    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:05:23.458077 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.458006    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:23.458645 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.458532    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:05:23.458645 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.458535    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:05:23.458645 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.458565    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:05:23.458645 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.458628    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:05:23.458645 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.458638    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:05:23.458853 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.458841    2577 factory.go:55] Registering systemd factory
Oct 30 19:05:23.458853 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.458852    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:05:23.459048 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.459032    2577 factory.go:153] Registering CRI-O factory
Oct 30 19:05:23.459048 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.459047    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:05:23.459164 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.459091    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:05:23.459164 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.459104    2577 factory.go:103] Registering Raw factory
Oct 30 19:05:23.459164 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.459118    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:05:23.460012 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.460001    2577 manager.go:319] Starting recovery of all containers
Oct 30 19:05:23.462418 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.462388    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-164.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:05:23.468982 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.468958    2577 manager.go:324] Recovery completed
Oct 30 19:05:23.473041 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.473028    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:23.474681 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.474665    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:23.474757 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.474695    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:23.474757 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.474709    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:23.475196 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.475184    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:23.475196 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.475189    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:05:23.475308 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.475198    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:05:23.475308 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.475212    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:05:23.475835 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.475822    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:23.475889 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.475847    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:23.475889 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.475856    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:23.476258 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.476241    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7t6jb"
Oct 30 19:05:23.476372 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.476268    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:05:23.477238 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.477226    2577 policy_none.go:50] "Start"
Oct 30 19:05:23.477349 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.477241    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:05:23.477349 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.477254    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:05:23.479592 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.479581    2577 policy_none.go:44] "Start"
Oct 30 19:05:23.480614 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.480539    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-164.us-east-2.compute.internal.18e364164777c969  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-164.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-164.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-164.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:05:23.474680169 +0000 UTC m=+0.429811588,LastTimestamp:2026-10-30 19:05:23.474680169 +0000 UTC m=+0.429811588,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-164.us-east-2.compute.internal,}"
Oct 30 19:05:23.490003 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.489120    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7t6jb"
Oct 30 19:05:23.490159 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.490056    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-164.us-east-2.compute.internal.18e3641647781f78  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-164.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-164.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-164.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:05:23.4747022 +0000 UTC m=+0.429833624,LastTimestamp:2026-10-30 19:05:23.4747022 +0000 UTC m=+0.429833624,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-164.us-east-2.compute.internal,}"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.517897    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.522172    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.522203    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.522214    2577 server.go:87] "Starting device plugin registration server"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.522465    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.522477    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.522549    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.522629    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.522640    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.523078    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.523115    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.530043    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.531319    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.531342    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.531364    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:05:23.534863 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.531400    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:05:23.537051 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.536198    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:23.623077 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.623024    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:23.623911 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.623895    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:23.623995 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.623930    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:23.623995 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.623945    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:23.623995 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.623973    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.632350 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.632330    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"]
Oct 30 19:05:23.632416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.632384    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:23.633132 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.633115    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:23.633201 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.633148    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:23.633201 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.633161    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:23.634801 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.634787    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.634876 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.634810    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-164.us-east-2.compute.internal\": node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:23.635347 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.635335    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:23.635485 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.635469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.635532 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.635500    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:23.636014 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.635999    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:23.636014 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.636008    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:23.636141 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.636032    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:23.636141 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.636048    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:23.636141 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.636034    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:23.636141 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.636117    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:23.638174 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.638162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.638224 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.638187    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:23.638841 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.638823    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:23.638929 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.638856    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:23.638929 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.638872    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:23.670988 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.670969    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found" node="ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.673404 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.673387    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found" node="ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.673676 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.673658    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:23.760223 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.760206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/64f9ae4e88f69ebe611e7b19f38b4cc6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"64f9ae4e88f69ebe611e7b19f38b4cc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.760351 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.760228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.760351 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.760248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.774501 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.774484    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:23.860791 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.860752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.860791 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.860796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.860961 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.860817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/64f9ae4e88f69ebe611e7b19f38b4cc6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"64f9ae4e88f69ebe611e7b19f38b4cc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.860961 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.860877    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/64f9ae4e88f69ebe611e7b19f38b4cc6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"64f9ae4e88f69ebe611e7b19f38b4cc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.860961 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.860883    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.860961 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.860884    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.874895 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.874852    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:23.975738 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:23.975699    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:23.975738 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.975723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:23.978888 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:23.978870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:24.076429 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:24.076397    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:24.176917 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:24.176845    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:24.277424 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:24.277383    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:24.378100 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:24.378073    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:24.381181 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.381156    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:05:24.381333 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.381316    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:05:24.478210 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:24.478155    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:05:24.495343 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.495308    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:00:24 +0000 UTC" deadline="2028-07-26 01:52:56.568543024 +0000 UTC"
Oct 30 19:05:24.495343 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.495335    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15222h47m32.073210759s"
Oct 30 19:05:24.543343 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.543321    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:05:24.543487 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.543401    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:24.544317 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.544301    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:05:24.544415 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.544330    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:05:24.544415 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.544341    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:05:24.558249 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.558230    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:24.571956 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.571942    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:05:24.574053 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.574035    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:05:24.580627 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.580611    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:05:24.629343 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:24.629316    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f9ae4e88f69ebe611e7b19f38b4cc6.slice/crio-212f7152c441ccfa9301249e058e618d195e0747939018f1b8920fb518c32a63 WatchSource:0}: Error finding container 212f7152c441ccfa9301249e058e618d195e0747939018f1b8920fb518c32a63: Status 404 returned error can't find the container with id 212f7152c441ccfa9301249e058e618d195e0747939018f1b8920fb518c32a63
Oct 30 19:05:24.629875 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:24.629854    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08b19af43d4124fc88d192e2bced995.slice/crio-966f43d2e6e3083fd23b390f7f2671a59b0fcd6bef9029c8977239423567835e WatchSource:0}: Error finding container 966f43d2e6e3083fd23b390f7f2671a59b0fcd6bef9029c8977239423567835e: Status 404 returned error can't find the container with id 966f43d2e6e3083fd23b390f7f2671a59b0fcd6bef9029c8977239423567835e
Oct 30 19:05:24.635505 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.635490    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:05:24.793070 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:24.793001    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:25.058953 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:25.058883    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:26.444296 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.444270    2577 apiserver.go:52] "Watching apiserver"
Oct 30 19:05:26.451744 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.451715    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:05:26.452529 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.452348    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-24ddg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t","openshift-cluster-node-tuning-operator/tuned-b8m22","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-h9ktv","openshift-multus/multus-mz8rn","kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal","openshift-dns/node-resolver-g76xc","openshift-multus/network-metrics-daemon-882bp","openshift-network-diagnostics/network-check-target-9s7mb","openshift-ovn-kubernetes/ovnkube-node-cgjj4","openshift-image-registry/node-ca-cnsll","openshift-network-operator/iptables-alerter-29spv","kube-system/global-pull-secret-syncer-kj2xn"]
Oct 30 19:05:26.455677 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.455654    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:26.456396 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.456368    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:26.457839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.457813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.460819 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.460795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.462148 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.462132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.462994 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.462959    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:05:26.463090 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.463009    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:05:26.463512 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.463491    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:05:26.463857 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.463839    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xd9sf\""
Oct 30 19:05:26.463957 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.463889    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:05:26.463957 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.463916    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gkzrx\""
Oct 30 19:05:26.464055 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.464003    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:05:26.464471 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.464451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.465278 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.465259    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:05:26.465278 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.465275    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:05:26.465428 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.465298    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ngr9b\""
Oct 30 19:05:26.465428 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.465330    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:05:26.466863 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.466846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.468757 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.468722    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jks97\""
Oct 30 19:05:26.468757 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.468738    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:05:26.468887 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.468810    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:05:26.468887 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.468826    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:05:26.468887 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.468835    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:05:26.468887 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.468878    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:05:26.469330 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.469315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.469772 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.469753    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:05:26.469772 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.469757    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-m74bb\""
Oct 30 19:05:26.471771 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.471755    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.471990 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.471971    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:05:26.472061 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.472011    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:05:26.472121 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.472098    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:05:26.472335 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.472319    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xlxzb\""
Oct 30 19:05:26.472513 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.472495    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:05:26.472585 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.472574    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:05:26.472850 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.472834    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:05:26.474340 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.474323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:26.474430 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.474378    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:26.476096 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.476080    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:05:26.476554 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.476537    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:05:26.477214 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.476822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:26.479230 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.479212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.481133 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.481117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:26.481227 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.481175    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:26.487320 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2s8\" (UniqueName: \"kubernetes.io/projected/2ec7ed63-0fc7-4c17-a301-a4035c43feef-kube-api-access-hp2s8\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.487417 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.487417 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.487417 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-k8s-cni-cncf-io\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.487417 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-netns\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.487417 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.487417 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-bin\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-config\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-system-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-kubelet\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-etc-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-script-lib\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-host\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-tuned\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a69fe59-2039-43ff-a132-eeed44422412-host\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.487603 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-hostroot\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-ovn\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-system-cni-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-run\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-slash\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7pf\" (UniqueName: \"kubernetes.io/projected/336005f6-92be-46c1-aa77-3fe7ab10970e-kube-api-access-wb7pf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-etc-kubernetes\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-systemd\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.487977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-kubelet\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.488032 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-node-log\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-netd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-tmp\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cnibin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-bin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-daemon-config\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-registration-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szps7\" (UniqueName: \"kubernetes.io/projected/3e61d03d-c4b1-4ac3-8802-27d32191c403-kube-api-access-szps7\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cni-binary-copy\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-systemd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-log-socket\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-var-lib-kubelet\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-netns\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovn-node-metrics-cert\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.488416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdzq\" (UniqueName: \"kubernetes.io/projected/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-kube-api-access-6bdzq\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-conf-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-socket-dir-parent\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-multus-certs\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-device-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xthsn\" (UniqueName: \"kubernetes.io/projected/9a69fe59-2039-43ff-a132-eeed44422412-kube-api-access-xthsn\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d11698df-8a69-42d9-96a9-ce0d5a37b003-agent-certs\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5thr\" (UniqueName: \"kubernetes.io/projected/0413837b-14e3-4465-8a24-1bca4f394e06-kube-api-access-j5thr\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-conf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a69fe59-2039-43ff-a132-eeed44422412-serviceca\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmkv\" (UniqueName: \"kubernetes.io/projected/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-kube-api-access-vjmkv\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-host-slash\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5d7d\" (UniqueName: \"kubernetes.io/projected/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-kube-api-access-t5d7d\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-kubernetes\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-multus\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.489074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwlx\" (UniqueName: \"kubernetes.io/projected/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kube-api-access-9cwlx\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysconfig\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-sys\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-iptables-alerter-script\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-systemd-units\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.488953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-hosts-file\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-tmp-dir\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-sys-fs\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-var-lib-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-env-overrides\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-cnibin\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d11698df-8a69-42d9-96a9-ce0d5a37b003-konnectivity-ca\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-socket-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.489809 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.490625 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-modprobe-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.490625 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-binary-copy\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.490625 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.490625 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-lib-modules\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.490625 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.489789    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:05:26.502913 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.502888    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:05:26.507266 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.507230    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-28z9s"
Oct 30 19:05:26.519619 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.519601    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-28z9s"
Oct 30 19:05:26.555635 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.555548    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"64f9ae4e88f69ebe611e7b19f38b4cc6","Type":"ContainerStarted","Data":"212f7152c441ccfa9301249e058e618d195e0747939018f1b8920fb518c32a63"}
Oct 30 19:05:26.558665 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.558638    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"a08b19af43d4124fc88d192e2bced995","Type":"ContainerStarted","Data":"966f43d2e6e3083fd23b390f7f2671a59b0fcd6bef9029c8977239423567835e"}
Oct 30 19:05:26.563213 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.563190    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:05:26.563480 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.563389    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:05:26.563480 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.563414    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:05:26.563480 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.563457    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:05:26.563480 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.563477    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:05:26.563744 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.563629    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w2h58\""
Oct 30 19:05:26.563744 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.563706    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:05:26.563744 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.563743    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hsp67\""
Oct 30 19:05:26.573728 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.573714    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4z6gp\""
Oct 30 19:05:26.578521 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.578503    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:05:26.590544 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-socket-dir-parent\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.590642 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-multus-certs\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.590642 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-device-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.590642 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.590818 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590652    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-device-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.590818 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xthsn\" (UniqueName: \"kubernetes.io/projected/9a69fe59-2039-43ff-a132-eeed44422412-kube-api-access-xthsn\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.590818 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590717    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-multus-certs\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.590818 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-dbus\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:26.590818 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590744    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-socket-dir-parent\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.590818 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d11698df-8a69-42d9-96a9-ce0d5a37b003-agent-certs\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5thr\" (UniqueName: \"kubernetes.io/projected/0413837b-14e3-4465-8a24-1bca4f394e06-kube-api-access-j5thr\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-conf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a69fe59-2039-43ff-a132-eeed44422412-serviceca\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmkv\" (UniqueName: \"kubernetes.io/projected/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-kube-api-access-vjmkv\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-host-slash\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5d7d\" (UniqueName: \"kubernetes.io/projected/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-kube-api-access-t5d7d\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-kubernetes\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-multus\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwlx\" (UniqueName: \"kubernetes.io/projected/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kube-api-access-9cwlx\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysconfig\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-sys\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.591086 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-iptables-alerter-script\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-systemd-units\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591148    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591208    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-hosts-file\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591370    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysconfig\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591404    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a69fe59-2039-43ff-a132-eeed44422412-serviceca\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591425    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-host-slash\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-sys\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591592    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-kubernetes\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.591670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591626    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-multus\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.590842    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591834    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-systemd-units\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591840    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-conf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-hosts-file\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591875    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-tmp-dir\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-sys-fs\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-var-lib-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591983    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-sys-fs\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.591986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-env-overrides\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-cnibin\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d11698df-8a69-42d9-96a9-ce0d5a37b003-konnectivity-ca\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592038    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-var-lib-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-socket-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592097    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.592240 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592120    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-iptables-alerter-script\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592174    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-tmp-dir\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-socket-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592229    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-cnibin\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-modprobe-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-binary-copy\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592309    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592325    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-modprobe-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-lib-modules\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2s8\" (UniqueName: \"kubernetes.io/projected/2ec7ed63-0fc7-4c17-a301-a4035c43feef-kube-api-access-hp2s8\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592578    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-env-overrides\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592703    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-lib-modules\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592727    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-binary-copy\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.593016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592758    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d11698df-8a69-42d9-96a9-ce0d5a37b003-konnectivity-ca\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-k8s-cni-cncf-io\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-netns\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592929    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.592948    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-netns\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593021    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-k8s-cni-cncf-io\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-bin\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593103    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-bin\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-config\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-system-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593196    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-kubelet\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593262    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-kubelet\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593308    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-system-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-etc-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-script-lib\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593418    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-etc-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.593587 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-host\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-tuned\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a69fe59-2039-43ff-a132-eeed44422412-host\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593656    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-host\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-hostroot\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593722    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-ovn\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593767    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a69fe59-2039-43ff-a132-eeed44422412-host\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-system-cni-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593775    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-run\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593811    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-hostroot\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593816    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-ovn\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593822    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-run\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-system-cni-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.594360 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-slash\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593923    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-slash\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.593969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7pf\" (UniqueName: \"kubernetes.io/projected/336005f6-92be-46c1-aa77-3fe7ab10970e-kube-api-access-wb7pf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.594057    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-etc-kubernetes\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-systemd\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-kubelet\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594125    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-etc-kubernetes\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594093    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-node-log\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.594147    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:27.094127278 +0000 UTC m=+3.030847721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594165    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-node-log\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594191    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-systemd\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-netd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-tmp\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.595745 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594230    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-netd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594198    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-kubelet\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cnibin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-bin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-daemon-config\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594304    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cnibin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594308    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-bin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-registration-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szps7\" (UniqueName: \"kubernetes.io/projected/3e61d03d-c4b1-4ac3-8802-27d32191c403-kube-api-access-szps7\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cni-binary-copy\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-systemd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-log-socket\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-var-lib-kubelet\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-netns\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-kubelet-config\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovn-node-metrics-cert\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdzq\" (UniqueName: \"kubernetes.io/projected/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-kube-api-access-6bdzq\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-conf-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.596595 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594644    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-netns\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594648    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-var-lib-kubelet\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594822    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-script-lib\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594875    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-daemon-config\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.594931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-registration-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.595000    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-conf-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.595061    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-log-socket\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.595132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-config\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.595229    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-systemd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.595777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cni-binary-copy\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.595878    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d11698df-8a69-42d9-96a9-ce0d5a37b003-agent-certs\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.597121    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-tmp\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.597477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.597273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-tuned\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.599035 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.599004    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovn-node-metrics-cert\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.606200 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.606171    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:26.606289 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.606207    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:26.606289 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.606220    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nrhsk for pod openshift-network-diagnostics/network-check-target-9s7mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:26.606289 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.606272    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk podName:3cbe6234-bffe-4a9f-a543-8c509b08357e nodeName:}" failed. No retries permitted until 2026-10-30 19:05:27.10625556 +0000 UTC m=+3.042975996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nrhsk" (UniqueName: "kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk") pod "network-check-target-9s7mb" (UID: "3cbe6234-bffe-4a9f-a543-8c509b08357e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:26.608880 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.608853    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmkv\" (UniqueName: \"kubernetes.io/projected/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-kube-api-access-vjmkv\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.609051 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.609030    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5d7d\" (UniqueName: \"kubernetes.io/projected/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-kube-api-access-t5d7d\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.609114 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.609059    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5thr\" (UniqueName: \"kubernetes.io/projected/0413837b-14e3-4465-8a24-1bca4f394e06-kube-api-access-j5thr\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:26.611332 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.611285    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwlx\" (UniqueName: \"kubernetes.io/projected/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kube-api-access-9cwlx\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.613911 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.613335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2s8\" (UniqueName: \"kubernetes.io/projected/2ec7ed63-0fc7-4c17-a301-a4035c43feef-kube-api-access-hp2s8\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.614670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.614644    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szps7\" (UniqueName: \"kubernetes.io/projected/3e61d03d-c4b1-4ac3-8802-27d32191c403-kube-api-access-szps7\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.614670 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.614657    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xthsn\" (UniqueName: \"kubernetes.io/projected/9a69fe59-2039-43ff-a132-eeed44422412-kube-api-access-xthsn\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.615038 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.615015    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdzq\" (UniqueName: \"kubernetes.io/projected/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-kube-api-access-6bdzq\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.616912 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.616890    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7pf\" (UniqueName: \"kubernetes.io/projected/336005f6-92be-46c1-aa77-3fe7ab10970e-kube-api-access-wb7pf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.694984 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.694895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-kubelet-config\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:26.694984 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.694949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-dbus\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:26.695201 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.695018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:26.695201 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.695029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-kubelet-config\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:26.695201 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.695111    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-dbus\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:26.695201 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.695133    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:26.695201 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:26.695193    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret podName:797c4f9e-bf72-41af-a1df-f39270f68e1a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:27.195176476 +0000 UTC m=+3.131896914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret") pod "global-pull-secret-syncer-kj2xn" (UID: "797c4f9e-bf72-41af-a1df-f39270f68e1a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:26.773729 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.773680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:05:26.783176 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.783148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:05:26.783728 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:26.783699    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc72d076_0c85_4722_a5fe_3c20dcd0b0e6.slice/crio-0aa5593d5694eb972e93f7cc5e3448a8f2cfa0ab94a834102d7ad3574edb5a3e WatchSource:0}: Error finding container 0aa5593d5694eb972e93f7cc5e3448a8f2cfa0ab94a834102d7ad3574edb5a3e: Status 404 returned error can't find the container with id 0aa5593d5694eb972e93f7cc5e3448a8f2cfa0ab94a834102d7ad3574edb5a3e
Oct 30 19:05:26.792869 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:26.792839    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336005f6_92be_46c1_aa77_3fe7ab10970e.slice/crio-9ecf1efcbaf0f4f4f9dde457ab8b1e68e4429deed4c0162e7d2542176301a076 WatchSource:0}: Error finding container 9ecf1efcbaf0f4f4f9dde457ab8b1e68e4429deed4c0162e7d2542176301a076: Status 404 returned error can't find the container with id 9ecf1efcbaf0f4f4f9dde457ab8b1e68e4429deed4c0162e7d2542176301a076
Oct 30 19:05:26.798532 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.798507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:05:26.808670 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:26.808640    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a69fe59_2039_43ff_a132_eeed44422412.slice/crio-5520ce5889c47c068fb865cade61ffb9163be214d467761137a7e7d2365fafbe WatchSource:0}: Error finding container 5520ce5889c47c068fb865cade61ffb9163be214d467761137a7e7d2365fafbe: Status 404 returned error can't find the container with id 5520ce5889c47c068fb865cade61ffb9163be214d467761137a7e7d2365fafbe
Oct 30 19:05:26.818753 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.818723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mz8rn"
Oct 30 19:05:26.822315 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.822287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:05:26.830631 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:26.830607    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3e41d8_e9c9_49bb_af2b_3ccf8acab832.slice/crio-df1e6b61bd4fd7b4b0d8cd476636fef4dd175e3ea6524a7ae5016c684c59ef8a WatchSource:0}: Error finding container df1e6b61bd4fd7b4b0d8cd476636fef4dd175e3ea6524a7ae5016c684c59ef8a: Status 404 returned error can't find the container with id df1e6b61bd4fd7b4b0d8cd476636fef4dd175e3ea6524a7ae5016c684c59ef8a
Oct 30 19:05:26.832497 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:26.832063    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e61d03d_c4b1_4ac3_8802_27d32191c403.slice/crio-4ceae37138053d042352d72c74d79c13e0a4ad093fce13ba773093779b24ca17 WatchSource:0}: Error finding container 4ceae37138053d042352d72c74d79c13e0a4ad093fce13ba773093779b24ca17: Status 404 returned error can't find the container with id 4ceae37138053d042352d72c74d79c13e0a4ad093fce13ba773093779b24ca17
Oct 30 19:05:26.832641 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.832622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:05:26.842521 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.842499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:26.843056 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:26.843036    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50accc7_ce62_4f71_9a4a_aa128cfe10c4.slice/crio-8bff760d87968173196ea3e9a1fe4715736ae9884952a5494102135d1d1a8a5c WatchSource:0}: Error finding container 8bff760d87968173196ea3e9a1fe4715736ae9884952a5494102135d1d1a8a5c: Status 404 returned error can't find the container with id 8bff760d87968173196ea3e9a1fe4715736ae9884952a5494102135d1d1a8a5c
Oct 30 19:05:26.849875 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.849850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:26.851543 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:26.851508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:05:26.854336 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:26.854311    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11698df_8a69_42d9_96a9_ce0d5a37b003.slice/crio-f8c8a827085254904a7915c0e10b86573f5d1c7f52481ac02f5af35046310c49 WatchSource:0}: Error finding container f8c8a827085254904a7915c0e10b86573f5d1c7f52481ac02f5af35046310c49: Status 404 returned error can't find the container with id f8c8a827085254904a7915c0e10b86573f5d1c7f52481ac02f5af35046310c49
Oct 30 19:05:26.860544 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:26.860523    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec7ed63_0fc7_4c17_a301_a4035c43feef.slice/crio-0ab5ca8613b565b6e3fe91d5b4d5a1589fd58846713bda11f6c235b9307e34aa WatchSource:0}: Error finding container 0ab5ca8613b565b6e3fe91d5b4d5a1589fd58846713bda11f6c235b9307e34aa: Status 404 returned error can't find the container with id 0ab5ca8613b565b6e3fe91d5b4d5a1589fd58846713bda11f6c235b9307e34aa
Oct 30 19:05:27.098060 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.098027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:27.098213 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:27.098173    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:27.098249 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:27.098230    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:28.098217325 +0000 UTC m=+4.034937760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:27.198700 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.198658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:27.198839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.198723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:27.198839 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:27.198811    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:27.198839 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:27.198817    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:27.198839 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:27.198831    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:27.198839 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:27.198842    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nrhsk for pod openshift-network-diagnostics/network-check-target-9s7mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:27.199043 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:27.198874    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret podName:797c4f9e-bf72-41af-a1df-f39270f68e1a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:28.198860945 +0000 UTC m=+4.135581381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret") pod "global-pull-secret-syncer-kj2xn" (UID: "797c4f9e-bf72-41af-a1df-f39270f68e1a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:27.199043 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:27.198892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk podName:3cbe6234-bffe-4a9f-a543-8c509b08357e nodeName:}" failed. No retries permitted until 2026-10-30 19:05:28.198883153 +0000 UTC m=+4.135603578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrhsk" (UniqueName: "kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk") pod "network-check-target-9s7mb" (UID: "3cbe6234-bffe-4a9f-a543-8c509b08357e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:27.523802 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.523522    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:00:26 +0000 UTC" deadline="2028-06-23 10:07:07.661752645 +0000 UTC"
Oct 30 19:05:27.524470 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.523805    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14439h1m40.137954622s"
Oct 30 19:05:27.589848 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.589673    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cnsll" event={"ID":"9a69fe59-2039-43ff-a132-eeed44422412","Type":"ContainerStarted","Data":"5520ce5889c47c068fb865cade61ffb9163be214d467761137a7e7d2365fafbe"}
Oct 30 19:05:27.594175 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.594094    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b8m22" event={"ID":"336005f6-92be-46c1-aa77-3fe7ab10970e","Type":"ContainerStarted","Data":"9ecf1efcbaf0f4f4f9dde457ab8b1e68e4429deed4c0162e7d2542176301a076"}
Oct 30 19:05:27.603752 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.602851    2577 generic.go:356] "Generic (PLEG): container finished" podID="a08b19af43d4124fc88d192e2bced995" containerID="57b558318dd96fd8c462c97a798aea9ded01191d083d980a347bce8ef8bcc118" exitCode=0
Oct 30 19:05:27.603752 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.602924    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"a08b19af43d4124fc88d192e2bced995","Type":"ContainerDied","Data":"57b558318dd96fd8c462c97a798aea9ded01191d083d980a347bce8ef8bcc118"}
Oct 30 19:05:27.607036 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.606956    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t" event={"ID":"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0","Type":"ContainerStarted","Data":"658dab09f57ff6bef530c98dc853d56069401b335461734b2440ba313e6ac6c5"}
Oct 30 19:05:27.609665 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.609587    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" event={"ID":"2ec7ed63-0fc7-4c17-a301-a4035c43feef","Type":"ContainerStarted","Data":"0ab5ca8613b565b6e3fe91d5b4d5a1589fd58846713bda11f6c235b9307e34aa"}
Oct 30 19:05:27.618936 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.618912    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h9ktv" event={"ID":"3e61d03d-c4b1-4ac3-8802-27d32191c403","Type":"ContainerStarted","Data":"4ceae37138053d042352d72c74d79c13e0a4ad093fce13ba773093779b24ca17"}
Oct 30 19:05:27.621807 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.621781    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mz8rn" event={"ID":"be3e41d8-e9c9-49bb-af2b-3ccf8acab832","Type":"ContainerStarted","Data":"df1e6b61bd4fd7b4b0d8cd476636fef4dd175e3ea6524a7ae5016c684c59ef8a"}
Oct 30 19:05:27.628509 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.628484    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29spv" event={"ID":"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6","Type":"ContainerStarted","Data":"0aa5593d5694eb972e93f7cc5e3448a8f2cfa0ab94a834102d7ad3574edb5a3e"}
Oct 30 19:05:27.648076 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.648030    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"64f9ae4e88f69ebe611e7b19f38b4cc6","Type":"ContainerStarted","Data":"7c506c34fc646f3580e8bcb8fea0902f8a68f7d6bd0770e9f1a1129875d95f18"}
Oct 30 19:05:27.652750 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.652725    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24ddg" event={"ID":"d11698df-8a69-42d9-96a9-ce0d5a37b003","Type":"ContainerStarted","Data":"f8c8a827085254904a7915c0e10b86573f5d1c7f52481ac02f5af35046310c49"}
Oct 30 19:05:27.654459 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:27.654377    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g76xc" event={"ID":"a50accc7-ce62-4f71-9a4a-aa128cfe10c4","Type":"ContainerStarted","Data":"8bff760d87968173196ea3e9a1fe4715736ae9884952a5494102135d1d1a8a5c"}
Oct 30 19:05:28.105477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.104905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:28.105477 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.105064    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:28.105477 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.105125    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:30.105107121 +0000 UTC m=+6.041827545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:28.206328 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.205386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:28.206328 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.205467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:28.206328 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.205707    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:28.206328 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.205727    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:28.206328 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.205738    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nrhsk for pod openshift-network-diagnostics/network-check-target-9s7mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:28.206328 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.205793    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk podName:3cbe6234-bffe-4a9f-a543-8c509b08357e nodeName:}" failed. No retries permitted until 2026-10-30 19:05:30.2057737 +0000 UTC m=+6.142494125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrhsk" (UniqueName: "kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk") pod "network-check-target-9s7mb" (UID: "3cbe6234-bffe-4a9f-a543-8c509b08357e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:28.206328 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.206205    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:28.206328 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.206262    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret podName:797c4f9e-bf72-41af-a1df-f39270f68e1a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:30.206246341 +0000 UTC m=+6.142966767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret") pod "global-pull-secret-syncer-kj2xn" (UID: "797c4f9e-bf72-41af-a1df-f39270f68e1a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:28.524983 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.524714    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:00:26 +0000 UTC" deadline="2028-07-13 05:04:45.386345324 +0000 UTC"
Oct 30 19:05:28.524983 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.524769    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14913h59m16.861580113s"
Oct 30 19:05:28.554884 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.554860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:28.555033 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.554978    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:28.555364 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.555344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:28.555484 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.555448    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:28.555553 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.555508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:28.555600 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:28.555573    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:28.668395 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.668324    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"a08b19af43d4124fc88d192e2bced995","Type":"ContainerStarted","Data":"3f47b662d4261a67d3bda8c864bcfcc383c98f5749b16cae714bcffbbbd63ac5"}
Oct 30 19:05:28.685771 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:28.685714    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal" podStartSLOduration=3.685661621 podStartE2EDuration="3.685661621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:27.66936282 +0000 UTC m=+3.606083245" watchObservedRunningTime="2026-10-30 19:05:28.685661621 +0000 UTC m=+4.622382065"
Oct 30 19:05:30.120494 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:30.119829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:30.120494 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.120016    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:30.120494 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.120110    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:34.1200916 +0000 UTC m=+10.056812040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:30.222017 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:30.221156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:30.222017 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:30.221250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:30.222017 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.221396    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:30.222017 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.221412    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:30.222017 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.221424    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nrhsk for pod openshift-network-diagnostics/network-check-target-9s7mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:30.222017 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.221484    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk podName:3cbe6234-bffe-4a9f-a543-8c509b08357e nodeName:}" failed. No retries permitted until 2026-10-30 19:05:34.221465915 +0000 UTC m=+10.158186336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrhsk" (UniqueName: "kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk") pod "network-check-target-9s7mb" (UID: "3cbe6234-bffe-4a9f-a543-8c509b08357e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:30.222017 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.221891    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:30.222017 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.221942    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret podName:797c4f9e-bf72-41af-a1df-f39270f68e1a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:34.221927788 +0000 UTC m=+10.158648223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret") pod "global-pull-secret-syncer-kj2xn" (UID: "797c4f9e-bf72-41af-a1df-f39270f68e1a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:30.552829 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:30.552788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:30.552976 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.552917    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:30.553274 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:30.553262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:30.553380 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.553341    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:30.553443 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:30.553389    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:30.553495 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:30.553467    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:30.677288 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:30.677258    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="658a05489150d7963d2f5af1274e0d6ed15db43629d80bdb4d57e2836c8b5c3b" exitCode=0
Oct 30 19:05:30.677402 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:30.677313    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h9ktv" event={"ID":"3e61d03d-c4b1-4ac3-8802-27d32191c403","Type":"ContainerDied","Data":"658a05489150d7963d2f5af1274e0d6ed15db43629d80bdb4d57e2836c8b5c3b"}
Oct 30 19:05:30.702542 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:30.702489    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal" podStartSLOduration=5.702471368 podStartE2EDuration="5.702471368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:28.686554823 +0000 UTC m=+4.623275267" watchObservedRunningTime="2026-10-30 19:05:30.702471368 +0000 UTC m=+6.639191813"
Oct 30 19:05:31.684516 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:31.683663    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24ddg" event={"ID":"d11698df-8a69-42d9-96a9-ce0d5a37b003","Type":"ContainerStarted","Data":"e56c18eb81528de5b2020f095c80974085548060d09508d1dd889a9b9eeefdf6"}
Oct 30 19:05:32.552910 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:32.552881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:32.553076 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:32.553001    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:32.553150 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:32.553077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:32.553207 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:32.553159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:32.553260 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:32.553210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:32.553311 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:32.553276    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:32.693605 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:32.693571    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cnsll" event={"ID":"9a69fe59-2039-43ff-a132-eeed44422412","Type":"ContainerStarted","Data":"81ea1e68ce8ea9a662e2198c221e9dd65bbcbd5ba21579835678c101ddbd8a46"}
Oct 30 19:05:32.709096 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:32.709040    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cnsll" podStartSLOduration=3.8338887010000002 podStartE2EDuration="8.709021859s" totalImagesPullingTime="4.875133158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:32.708725048 +0000 UTC m=+8.645445492" watchObservedRunningTime="2026-10-30 19:05:32.709021859 +0000 UTC m=+8.645742306"
Oct 30 19:05:32.709268 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:32.709218    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-24ddg" podStartSLOduration=3.899337508 podStartE2EDuration="7.70921157s" totalImagesPullingTime="3.809874062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:31.701088623 +0000 UTC m=+7.637809067" watchObservedRunningTime="2026-10-30 19:05:32.70921157 +0000 UTC m=+8.645932012"
Oct 30 19:05:33.699422 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:33.699361    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t" event={"ID":"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0","Type":"ContainerStarted","Data":"11b12c4525efa34a4fc5e809783d23764b1f9686d258c819badbcb4832977836"}
Oct 30 19:05:34.005257 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.005169    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:34.007017 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.006724    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:34.154867 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.154236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:34.154867 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.154415    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:34.154867 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.154489    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:42.154470294 +0000 UTC m=+18.091190763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:34.255734 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.255246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:34.255734 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.255324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:34.255734 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.255473    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:34.255734 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.255491    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:34.255734 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.255504    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nrhsk for pod openshift-network-diagnostics/network-check-target-9s7mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:34.255734 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.255556    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk podName:3cbe6234-bffe-4a9f-a543-8c509b08357e nodeName:}" failed. No retries permitted until 2026-10-30 19:05:42.255538772 +0000 UTC m=+18.192259214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrhsk" (UniqueName: "kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk") pod "network-check-target-9s7mb" (UID: "3cbe6234-bffe-4a9f-a543-8c509b08357e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:34.255734 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.255639    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:34.255734 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.255680    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret podName:797c4f9e-bf72-41af-a1df-f39270f68e1a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:42.255666577 +0000 UTC m=+18.192387015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret") pod "global-pull-secret-syncer-kj2xn" (UID: "797c4f9e-bf72-41af-a1df-f39270f68e1a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:34.446906 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.446873    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:05:34.546446 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.546343    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:05:34.446903185Z","UUID":"e10e5a58-0657-4fc3-9f45-e5a50cc61668","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:05:34.547811 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.547786    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:05:34.547909 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.547819    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:05:34.554667 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.553945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:34.554667 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.554043    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:34.554667 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.554402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:34.554667 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.554508    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:34.554667 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.554566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:34.554667 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:34.554638    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:34.704076 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.704004    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mz8rn" event={"ID":"be3e41d8-e9c9-49bb-af2b-3ccf8acab832","Type":"ContainerStarted","Data":"16639853adb4fc0bc4c8da41d53c3a8a69404968314bae991cb85f048633b3cf"}
Oct 30 19:05:34.709876 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.709560    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g76xc" event={"ID":"a50accc7-ce62-4f71-9a4a-aa128cfe10c4","Type":"ContainerStarted","Data":"15e5c2a95e5eca1ce635c9c5c5b26de3693d77468882e713a8d9900ffc91637b"}
Oct 30 19:05:34.713416 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.713391    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t" event={"ID":"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0","Type":"ContainerStarted","Data":"cd95a7b9259193d705b4aa332cdc9d05488d19903b2ea8ececb15383d207d27b"}
Oct 30 19:05:34.726489 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.726448    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mz8rn" podStartSLOduration=3.32891013 podStartE2EDuration="10.726436187s" totalImagesPullingTime="7.397526057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:34.725787005 +0000 UTC m=+10.662507449" watchObservedRunningTime="2026-10-30 19:05:34.726436187 +0000 UTC m=+10.663156630"
Oct 30 19:05:34.740918 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:34.740870    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-g76xc" podStartSLOduration=3.088145362 podStartE2EDuration="10.740850466s" totalImagesPullingTime="7.652705104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:34.740501263 +0000 UTC m=+10.677221706" watchObservedRunningTime="2026-10-30 19:05:34.740850466 +0000 UTC m=+10.677570910"
Oct 30 19:05:35.720808 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:35.720583    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="1bff2c2ae6d66f7fb1ea6524984adcaa52f51b85f52d023e457a33840c6f0317" exitCode=0
Oct 30 19:05:35.721351 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:35.720631    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h9ktv" event={"ID":"3e61d03d-c4b1-4ac3-8802-27d32191c403","Type":"ContainerDied","Data":"1bff2c2ae6d66f7fb1ea6524984adcaa52f51b85f52d023e457a33840c6f0317"}
Oct 30 19:05:35.725091 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:35.725039    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t" event={"ID":"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0","Type":"ContainerStarted","Data":"8b7e9600387c021bd8f9cb98672d3eee22c0284f96eb53fa3b8cff0c7a0ed1df"}
Oct 30 19:05:36.550764 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:36.550735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:36.550891 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:36.550772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:36.550891 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:36.550745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:36.550891 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:36.550841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:36.551048 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:36.550928    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:36.551048 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:36.550974    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:36.728830 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:36.728791    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29spv" event={"ID":"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6","Type":"ContainerStarted","Data":"510c3c781c03b8fbc25dd1e07b10aab640171abc514f5aac2340152a1d88afc7"}
Oct 30 19:05:36.744888 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:36.744837    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-29spv" podStartSLOduration=4.767490373 podStartE2EDuration="12.744824341s" totalImagesPullingTime="7.977333968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:36.744541372 +0000 UTC m=+12.681261816" watchObservedRunningTime="2026-10-30 19:05:36.744824341 +0000 UTC m=+12.681544784"
Oct 30 19:05:36.745035 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:36.745010    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t" podStartSLOduration=3.515484513 podStartE2EDuration="11.744999622s" totalImagesPullingTime="8.229515109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:35.763749 +0000 UTC m=+11.700469445" watchObservedRunningTime="2026-10-30 19:05:36.744999622 +0000 UTC m=+12.681720065"
Oct 30 19:05:37.733916 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:37.733882    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="0d9f26846bf8d876f4cb9ec83cc19495066ed24ae9f32ee8436dd52749fb3378" exitCode=0
Oct 30 19:05:37.734502 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:37.733974    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h9ktv" event={"ID":"3e61d03d-c4b1-4ac3-8802-27d32191c403","Type":"ContainerDied","Data":"0d9f26846bf8d876f4cb9ec83cc19495066ed24ae9f32ee8436dd52749fb3378"}
Oct 30 19:05:37.736400 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:37.736252    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b8m22" event={"ID":"336005f6-92be-46c1-aa77-3fe7ab10970e","Type":"ContainerStarted","Data":"9eb852155480c1aea3075e4cc644bd04eec77297e107f564b62e21adc47c1809"}
Oct 30 19:05:37.774671 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:37.774613    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b8m22" podStartSLOduration=3.428549668 podStartE2EDuration="13.774594517s" totalImagesPullingTime="10.346044849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:37.77387132 +0000 UTC m=+13.710591762" watchObservedRunningTime="2026-10-30 19:05:37.774594517 +0000 UTC m=+13.711314959"
Oct 30 19:05:38.553275 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:38.553240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:38.553439 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:38.553312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:38.553439 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:38.553341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:38.553548 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:38.553456    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:38.553548 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:38.553512    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:38.553627 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:38.553599    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:39.743332 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:39.743105    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="01a7238a5e0c089bb848e5c8b172f7817eec47b98adf7ba7d10643e5cf521b21" exitCode=0
Oct 30 19:05:39.743905 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:39.743203    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h9ktv" event={"ID":"3e61d03d-c4b1-4ac3-8802-27d32191c403","Type":"ContainerDied","Data":"01a7238a5e0c089bb848e5c8b172f7817eec47b98adf7ba7d10643e5cf521b21"}
Oct 30 19:05:40.554401 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:40.554369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:40.554554 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:40.554449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:40.554596 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:40.554562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:40.554596 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:40.554576    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:40.554723 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:40.554653    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:40.554780 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:40.554744    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:41.749707 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:41.749496    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" event={"ID":"2ec7ed63-0fc7-4c17-a301-a4035c43feef","Type":"ContainerStarted","Data":"1903e16fe327dc9ab9befbb073125fa4892aedd2760027ffa5a6091efa53db49"}
Oct 30 19:05:41.750167 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:41.749757    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" event={"ID":"2ec7ed63-0fc7-4c17-a301-a4035c43feef","Type":"ContainerStarted","Data":"ab4c3feb62a5a6869cba75334d03f83a5b64cf0465ba89de74e083353f91afc3"}
Oct 30 19:05:41.750167 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:41.749792    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" event={"ID":"2ec7ed63-0fc7-4c17-a301-a4035c43feef","Type":"ContainerStarted","Data":"eef1db326190c37a607c4b7e9acdc0fa86ed8730ae9e1d43969213d96fa5f9ed"}
Oct 30 19:05:41.750167 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:41.749823    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" event={"ID":"2ec7ed63-0fc7-4c17-a301-a4035c43feef","Type":"ContainerStarted","Data":"0fb0c4e5e8d270647bb948621dfb9da72301d253c14126f8d8d9cedc243fc98d"}
Oct 30 19:05:41.750167 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:41.749865    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" event={"ID":"2ec7ed63-0fc7-4c17-a301-a4035c43feef","Type":"ContainerStarted","Data":"50d2767ee0b3482c4197415897c7d63ddb45fc83f58d70d28b5a7739f791f63f"}
Oct 30 19:05:41.750167 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:41.749926    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" event={"ID":"2ec7ed63-0fc7-4c17-a301-a4035c43feef","Type":"ContainerStarted","Data":"2fa496fa5917d611af6eade360d40f0322079256ad622432ba742ee55af702a7"}
Oct 30 19:05:41.752395 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:41.752332    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="be350b506d74dbb956d05949e613759050195985c812feeb4fb0a767b608128a" exitCode=0
Oct 30 19:05:41.752395 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:41.752363    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h9ktv" event={"ID":"3e61d03d-c4b1-4ac3-8802-27d32191c403","Type":"ContainerDied","Data":"be350b506d74dbb956d05949e613759050195985c812feeb4fb0a767b608128a"}
Oct 30 19:05:42.215818 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:42.215733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:42.215986 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.215856    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:42.215986 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.215904    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:58.21589279 +0000 UTC m=+34.152613224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:05:42.316900 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:42.316868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:42.317016 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:42.316916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:42.317016 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.317007    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:42.317100 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.317018    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:05:42.317100 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.317036    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:05:42.317100 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.317044    2577 projected.go:196] Error preparing data for projected volume kube-api-access-nrhsk for pod openshift-network-diagnostics/network-check-target-9s7mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:42.317100 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.317051    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret podName:797c4f9e-bf72-41af-a1df-f39270f68e1a nodeName:}" failed. No retries permitted until 2026-10-30 19:05:58.317039249 +0000 UTC m=+34.253759688 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret") pod "global-pull-secret-syncer-kj2xn" (UID: "797c4f9e-bf72-41af-a1df-f39270f68e1a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:05:42.317100 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.317081    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk podName:3cbe6234-bffe-4a9f-a543-8c509b08357e nodeName:}" failed. No retries permitted until 2026-10-30 19:05:58.317069873 +0000 UTC m=+34.253790297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrhsk" (UniqueName: "kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk") pod "network-check-target-9s7mb" (UID: "3cbe6234-bffe-4a9f-a543-8c509b08357e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:05:42.553057 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:42.552987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:42.553177 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:42.553111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:42.553177 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.553117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:42.553177 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:42.553136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:42.553272 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.553216    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:42.553301 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:42.553277    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:42.756833 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:42.756796    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="1768dda1663bcbebde47fdf5e26cca988c55bac174b4af9c4db90256ed4db797" exitCode=0
Oct 30 19:05:42.757243 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:42.756855    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h9ktv" event={"ID":"3e61d03d-c4b1-4ac3-8802-27d32191c403","Type":"ContainerDied","Data":"1768dda1663bcbebde47fdf5e26cca988c55bac174b4af9c4db90256ed4db797"}
Oct 30 19:05:43.762654 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:43.762623    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" event={"ID":"2ec7ed63-0fc7-4c17-a301-a4035c43feef","Type":"ContainerStarted","Data":"dbfeac9a88be44e7ccb434fd953ebcf91d2c36d45a2dc1bb236ed40a60eec202"}
Oct 30 19:05:43.765817 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:43.765792    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h9ktv" event={"ID":"3e61d03d-c4b1-4ac3-8802-27d32191c403","Type":"ContainerStarted","Data":"8fa55964c9bbc5c5bc4bb72f64a4b0418ee3066c8d901227f9849b8b0e218314"}
Oct 30 19:05:43.790881 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:43.790828    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h9ktv" podStartSLOduration=8.749469759 podStartE2EDuration="19.790809996s" totalImagesPullingTime="11.041340237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:43.790424431 +0000 UTC m=+19.727144873" watchObservedRunningTime="2026-10-30 19:05:43.790809996 +0000 UTC m=+19.727530439"
Oct 30 19:05:44.552922 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:44.552895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:44.552922 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:44.552919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:44.553147 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:44.552934    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:44.553147 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:44.552992    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:44.553147 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:44.553095    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:44.553147 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:44.553140    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:46.550263 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:46.550230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:46.550843 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:46.550229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:46.550843 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:46.550335    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:46.550843 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:46.550239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:46.550843 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:46.550415    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:46.550843 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:46.550502    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:46.775570 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:46.775540    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" event={"ID":"2ec7ed63-0fc7-4c17-a301-a4035c43feef","Type":"ContainerStarted","Data":"7d5f83cebd7d8f05013fdc3477e76bd28a7cb0dd8c39adab83b77cc5cb527c06"}
Oct 30 19:05:46.775805 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:46.775787    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:46.793000 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:46.792857    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:46.805906 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:46.805834    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4" podStartSLOduration=8.394689598 podStartE2EDuration="22.805823988s" totalImagesPullingTime="14.41113439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:46.804965605 +0000 UTC m=+22.741686048" watchObservedRunningTime="2026-10-30 19:05:46.805823988 +0000 UTC m=+22.742544431"
Oct 30 19:05:47.630152 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:47.630124    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9s7mb"]
Oct 30 19:05:47.630651 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:47.630263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:47.630651 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:47.630346    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:47.632967 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:47.632943    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kj2xn"]
Oct 30 19:05:47.633067 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:47.633023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:47.633100 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:47.633090    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:47.635564 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:47.635545    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-882bp"]
Oct 30 19:05:47.635632 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:47.635626    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:47.635731 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:47.635716    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:47.777369 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:47.777343    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:05:47.777737 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:47.777719    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:47.793403 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:47.793382    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:48.779089 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:48.779067    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:05:49.549994 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:49.549961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:49.549994 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:49.549992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:49.550248 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:49.549994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:49.550248 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:49.550064    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:49.550248 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:49.550130    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:49.550248 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:49.550198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:49.781733 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:49.781706    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:05:51.550800 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:51.550770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:51.551268 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:51.550872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:51.551268 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:51.550880    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:51.551268 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:51.550881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:51.551268 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:51.550959    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:51.551268 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:51.551050    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:53.550179 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.550137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:53.550608 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.550212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:53.550608 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.550298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:53.550608 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:53.550304    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9s7mb" podUID="3cbe6234-bffe-4a9f-a543-8c509b08357e"
Oct 30 19:05:53.550608 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:53.550395    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-882bp" podUID="0413837b-14e3-4465-8a24-1bca4f394e06"
Oct 30 19:05:53.550608 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:53.550488    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj2xn" podUID="797c4f9e-bf72-41af-a1df-f39270f68e1a"
Oct 30 19:05:53.791832 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.791809    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeReady"
Oct 30 19:05:53.791980 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.791942    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:05:53.827924 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.827851    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-98bt4"]
Oct 30 19:05:53.830348 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.830328    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2xnmc"]
Oct 30 19:05:53.830479 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.830460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-98bt4"
Oct 30 19:05:53.833657 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.833628    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jkwnj"]
Oct 30 19:05:53.833884 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.833866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2xnmc"
Oct 30 19:05:53.838209 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.838191    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bd99d66bc-r6hq2"]
Oct 30 19:05:53.838351 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.838332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jkwnj"
Oct 30 19:05:53.841033 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.841017    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"]
Oct 30 19:05:53.841139 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.841121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:53.842073 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.842046    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.842351 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.842331    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.843824 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.843808    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"]
Oct 30 19:05:53.843954 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.843940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:53.853206 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.853159    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:05:53.853325 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.853317    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vt4hq\""
Oct 30 19:05:53.854176 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.854146    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9b8n4\""
Oct 30 19:05:53.854292 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.854177    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:05:53.855130 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.855107    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:05:53.855672 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.855644    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"]
Oct 30 19:05:53.855978 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.855955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:53.861451 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.860529    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:53.861451 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.860600    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"]
Oct 30 19:05:53.861887 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.861795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"
Oct 30 19:05:53.862258 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.862240    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:05:53.864706 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.864668    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fskzc"]
Oct 30 19:05:53.869314 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.864849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:53.871764 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.871745    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"]
Oct 30 19:05:53.872290 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.872271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:53.876048 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.875143    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:05:53.876048 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.875204    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"]
Oct 30 19:05:53.876286 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.876269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:05:53.879654 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.879634    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-ddbc459d5-jmrh9"]
Oct 30 19:05:53.879811 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.879795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:53.883964 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.883941    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-775fb78d4b-vms7p"]
Oct 30 19:05:53.884477 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.884455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:53.890023 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.890002    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-h6hw8"]
Oct 30 19:05:53.890117 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.890096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:53.893055 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.893037    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"]
Oct 30 19:05:53.893346 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.893334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:53.895861 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.895842    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2xnmc"]
Oct 30 19:05:53.895947 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.895870    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jkwnj"]
Oct 30 19:05:53.895947 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.895890    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4dcpr"]
Oct 30 19:05:53.896017 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.895953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"
Oct 30 19:05:53.898907 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.898887    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"]
Oct 30 19:05:53.898999 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.898912    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-98bt4"]
Oct 30 19:05:53.898999 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.898925    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"]
Oct 30 19:05:53.898999 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.898938    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fskzc"]
Oct 30 19:05:53.898999 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.898952    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nb5x6"]
Oct 30 19:05:53.898999 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.898996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:53.901744 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901727    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bd99d66bc-r6hq2"]
Oct 30 19:05:53.901839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901751    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"]
Oct 30 19:05:53.901839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901770    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-h6hw8"]
Oct 30 19:05:53.901839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901784    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddbc459d5-jmrh9"]
Oct 30 19:05:53.901839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901797    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"]
Oct 30 19:05:53.901839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901810    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-775fb78d4b-vms7p"]
Oct 30 19:05:53.901839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:05:53.901839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901824    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"]
Oct 30 19:05:53.901839 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901837    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"]
Oct 30 19:05:53.902258 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901853    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nb5x6"]
Oct 30 19:05:53.902258 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901865    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4dcpr"]
Oct 30 19:05:53.902258 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.901881    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"]
Oct 30 19:05:53.904418 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.904397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn4h5\" (UniqueName: \"kubernetes.io/projected/c91b7f8f-3427-4d04-a5bd-31792aed5075-kube-api-access-gn4h5\") pod \"kas-connection-checker-685895c7c9-jkwnj\" (UID: \"c91b7f8f-3427-4d04-a5bd-31792aed5075\") " pod="kube-system/kas-connection-checker-685895c7c9-jkwnj"
Oct 30 19:05:53.904510 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.904460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlpzx\" (UniqueName: \"kubernetes.io/projected/3c9100be-82b2-466c-833c-3ed05703fba2-kube-api-access-wlpzx\") pod \"kas-connection-checker-685895c7c9-98bt4\" (UID: \"3c9100be-82b2-466c-833c-3ed05703fba2\") " pod="kube-system/kas-connection-checker-685895c7c9-98bt4"
Oct 30 19:05:53.904561 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.904532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8fjb\" (UniqueName: \"kubernetes.io/projected/6a4b6b51-385b-4e07-9eae-5d57cd734d39-kube-api-access-f8fjb\") pod \"kas-connection-checker-685895c7c9-2xnmc\" (UID: \"6a4b6b51-385b-4e07-9eae-5d57cd734d39\") " pod="kube-system/kas-connection-checker-685895c7c9-2xnmc"
Oct 30 19:05:53.942402 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942369    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.942514 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942400    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.942514 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942407    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.942514 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942430    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:05:53.942514 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942462    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.942658 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942637    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.942748 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942731    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.942939 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942789    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.942939 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942888    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.943079 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942967    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:05:53.943079 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942989    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.943079 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.942994    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.943079 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943011    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:05:53.943079 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943025    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:05:53.943079 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943014    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:05:53.943079 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943057    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:05:53.943079 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943067    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943144    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943156    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943207    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943210    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943278    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943280    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943366    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943387    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943396    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943409    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.943530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.943493    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.945180 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.945163    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:05:53.947403 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.947361    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:05:53.947654 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.947629    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:05:53.947760 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.947657    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:05:53.950066 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.949960    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:05:53.950066 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.949983    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:05:53.950066 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.949997    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:05:53.950066 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.949997    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:05:53.950066 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950044    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:05:53.950364 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950091    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:05:53.950364 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950249    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:05:53.950364 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950276    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:05:53.950364 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950304    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hfhz5\""
Oct 30 19:05:53.950364 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950330    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:05:53.950592 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950399    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6h85f\""
Oct 30 19:05:53.950592 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950409    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-frsmn\""
Oct 30 19:05:53.950592 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950441    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:05:53.950592 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950453    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:05:53.950592 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950488    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:05:53.950592 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950579    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ktwjf\""
Oct 30 19:05:53.950592 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950595    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7gl8m\""
Oct 30 19:05:53.950873 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950615    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-trztf\""
Oct 30 19:05:53.950873 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950641    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rdn2g\""
Oct 30 19:05:53.950873 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950657    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-stgll\""
Oct 30 19:05:53.950873 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950658    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ml89v\""
Oct 30 19:05:53.950873 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950655    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wfls2\""
Oct 30 19:05:53.950873 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.950714    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xxr6f\""
Oct 30 19:05:53.967827 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.967810    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:53.968320 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:53.968306    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:05:54.005182 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.004987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5250b4fe-4b08-4a7a-bfc7-d7343f202585-tmp-dir\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.005437 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:05:54.005530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.005628 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.005754 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk967\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.005754 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-stats-auth\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.005936 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:54.005936 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.005936 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6996469c-158c-46af-9e06-a89a9f7137c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:54.005936 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2b4\" (UniqueName: \"kubernetes.io/projected/1aaa292a-397b-4c16-a897-f4df8a4af32f-kube-api-access-pf2b4\") pod \"network-check-source-6f78d59f99-6d9pz\" (UID: \"1aaa292a-397b-4c16-a897-f4df8a4af32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"
Oct 30 19:05:54.005936 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-snapshots\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.006232 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9e20809-cbf6-491c-b6e9-371655cbd3c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:05:54.006232 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.005989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/151fec2f-192b-4af4-9d4e-10a790100cb9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.006232 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmbk\" (UniqueName: \"kubernetes.io/projected/45274ad9-273b-467a-8bf1-27a876267294-kube-api-access-vtmbk\") pod \"volume-data-source-validator-58c9f5f46d-fndkp\" (UID: \"45274ad9-273b-467a-8bf1-27a876267294\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"
Oct 30 19:05:54.006232 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pfd\" (UniqueName: \"kubernetes.io/projected/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-kube-api-access-55pfd\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.006232 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.006232 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.006232 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.006232 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrvr\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.006232 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmqs8\" (UniqueName: \"kubernetes.io/projected/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-kube-api-access-kmqs8\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.006633 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/664daa2d-471e-4380-bfdc-73df3a83ab3e-serving-cert\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.006633 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.006633 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-default-certificate\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.006633 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:54.006633 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-config\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.006633 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.006633 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6js6\" (UniqueName: \"kubernetes.io/projected/5250b4fe-4b08-4a7a-bfc7-d7343f202585-kube-api-access-v6js6\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.006633 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.006496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-trusted-ca\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.007501 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.007480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5250b4fe-4b08-4a7a-bfc7-d7343f202585-config-volume\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.007585 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.007525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgl6d\" (UniqueName: \"kubernetes.io/projected/664daa2d-471e-4380-bfdc-73df3a83ab3e-kube-api-access-wgl6d\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.007585 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.007555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk78m\" (UniqueName: \"kubernetes.io/projected/a09e9550-86c1-474b-9774-50dd9189f3fa-kube-api-access-xk78m\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:05:54.007723 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.007612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4h5\" (UniqueName: \"kubernetes.io/projected/c91b7f8f-3427-4d04-a5bd-31792aed5075-kube-api-access-gn4h5\") pod \"kas-connection-checker-685895c7c9-jkwnj\" (UID: \"c91b7f8f-3427-4d04-a5bd-31792aed5075\") " pod="kube-system/kas-connection-checker-685895c7c9-jkwnj"
Oct 30 19:05:54.007723 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.007640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-tmp\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.007723 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.007673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.007896 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.007721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46s4\" (UniqueName: \"kubernetes.io/projected/8c6e8d41-096c-4121-b9c9-a147a7ee010d-kube-api-access-m46s4\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.007896 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.007749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09e9550-86c1-474b-9774-50dd9189f3fa-cert\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:05:54.008073 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.008051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-service-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.008181 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.008148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hst\" (UniqueName: \"kubernetes.io/projected/6996469c-158c-46af-9e06-a89a9f7137c0-kube-api-access-55hst\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:54.008569 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.008549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-serving-cert\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.008626 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.008604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.008721 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.008652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.008721 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.008704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/c0a95351-6d86-4411-903e-3514b69f6bb7-kube-api-access-bb4ct\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:54.008842 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.008753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlpzx\" (UniqueName: \"kubernetes.io/projected/3c9100be-82b2-466c-833c-3ed05703fba2-kube-api-access-wlpzx\") pod \"kas-connection-checker-685895c7c9-98bt4\" (UID: \"3c9100be-82b2-466c-833c-3ed05703fba2\") " pod="kube-system/kas-connection-checker-685895c7c9-98bt4"
Oct 30 19:05:54.009068 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.009046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.009957 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.009938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:54.010377 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fjb\" (UniqueName: \"kubernetes.io/projected/6a4b6b51-385b-4e07-9eae-5d57cd734d39-kube-api-access-f8fjb\") pod \"kas-connection-checker-685895c7c9-2xnmc\" (UID: \"6a4b6b51-385b-4e07-9eae-5d57cd734d39\") " pod="kube-system/kas-connection-checker-685895c7c9-2xnmc"
Oct 30 19:05:54.010421 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.010468 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-serving-cert\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.010530 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.010567 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-config\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.010607 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.010661 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151fec2f-192b-4af4-9d4e-10a790100cb9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.010739 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.010776 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6tx\" (UniqueName: \"kubernetes.io/projected/151fec2f-192b-4af4-9d4e-10a790100cb9-kube-api-access-rf6tx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.010776 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.010844 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.010794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.021951 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.021847    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fjb\" (UniqueName: \"kubernetes.io/projected/6a4b6b51-385b-4e07-9eae-5d57cd734d39-kube-api-access-f8fjb\") pod \"kas-connection-checker-685895c7c9-2xnmc\" (UID: \"6a4b6b51-385b-4e07-9eae-5d57cd734d39\") " pod="kube-system/kas-connection-checker-685895c7c9-2xnmc"
Oct 30 19:05:54.022015 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.021930    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4h5\" (UniqueName: \"kubernetes.io/projected/c91b7f8f-3427-4d04-a5bd-31792aed5075-kube-api-access-gn4h5\") pod \"kas-connection-checker-685895c7c9-jkwnj\" (UID: \"c91b7f8f-3427-4d04-a5bd-31792aed5075\") " pod="kube-system/kas-connection-checker-685895c7c9-jkwnj"
Oct 30 19:05:54.022015 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.021930    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlpzx\" (UniqueName: \"kubernetes.io/projected/3c9100be-82b2-466c-833c-3ed05703fba2-kube-api-access-wlpzx\") pod \"kas-connection-checker-685895c7c9-98bt4\" (UID: \"3c9100be-82b2-466c-833c-3ed05703fba2\") " pod="kube-system/kas-connection-checker-685895c7c9-98bt4"
Oct 30 19:05:54.112155 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:54.112155 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.112155 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6996469c-158c-46af-9e06-a89a9f7137c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2b4\" (UniqueName: \"kubernetes.io/projected/1aaa292a-397b-4c16-a897-f4df8a4af32f-kube-api-access-pf2b4\") pod \"network-check-source-6f78d59f99-6d9pz\" (UID: \"1aaa292a-397b-4c16-a897-f4df8a4af32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-snapshots\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9e20809-cbf6-491c-b6e9-371655cbd3c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.112234    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/151fec2f-192b-4af4-9d4e-10a790100cb9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmbk\" (UniqueName: \"kubernetes.io/projected/45274ad9-273b-467a-8bf1-27a876267294-kube-api-access-vtmbk\") pod \"volume-data-source-validator-58c9f5f46d-fndkp\" (UID: \"45274ad9-273b-467a-8bf1-27a876267294\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.112302    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls podName:6996469c-158c-46af-9e06-a89a9f7137c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:54.612285024 +0000 UTC m=+30.549005452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rmqbj" (UID: "6996469c-158c-46af-9e06-a89a9f7137c0") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pfd\" (UniqueName: \"kubernetes.io/projected/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-kube-api-access-55pfd\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.112379 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrvr\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmqs8\" (UniqueName: \"kubernetes.io/projected/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-kube-api-access-kmqs8\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/664daa2d-471e-4380-bfdc-73df3a83ab3e-serving-cert\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-default-certificate\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-config\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.112633    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.112660    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-775fb78d4b-vms7p: secret "image-registry-tls" not found
Oct 30 19:05:54.112879 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.113561 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6js6\" (UniqueName: \"kubernetes.io/projected/5250b4fe-4b08-4a7a-bfc7-d7343f202585-kube-api-access-v6js6\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.113561 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112913    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-snapshots\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.113561 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112937    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/151fec2f-192b-4af4-9d4e-10a790100cb9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.113561 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113179    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.113561 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113313    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6996469c-158c-46af-9e06-a89a9f7137c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:54.113561 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.113357    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls podName:8b9d8478-9a05-4d2c-9210-a08764728647 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:54.613339117 +0000 UTC m=+30.550059553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls") pod "image-registry-775fb78d4b-vms7p" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647") : secret "image-registry-tls" not found
Oct 30 19:05:54.113890 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.112921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-trusted-ca\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.113890 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5250b4fe-4b08-4a7a-bfc7-d7343f202585-config-volume\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.113890 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113757    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-trusted-ca\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.113890 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgl6d\" (UniqueName: \"kubernetes.io/projected/664daa2d-471e-4380-bfdc-73df3a83ab3e-kube-api-access-wgl6d\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.113890 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk78m\" (UniqueName: \"kubernetes.io/projected/a09e9550-86c1-474b-9774-50dd9189f3fa-kube-api-access-xk78m\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:05:54.113890 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-tmp\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.113890 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46s4\" (UniqueName: \"kubernetes.io/projected/8c6e8d41-096c-4121-b9c9-a147a7ee010d-kube-api-access-m46s4\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09e9550-86c1-474b-9774-50dd9189f3fa-cert\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113935    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-service-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hst\" (UniqueName: \"kubernetes.io/projected/6996469c-158c-46af-9e06-a89a9f7137c0-kube-api-access-55hst\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-serving-cert\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/c0a95351-6d86-4411-903e-3514b69f6bb7-kube-api-access-bb4ct\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:54.114215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-serving-cert\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114264    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5250b4fe-4b08-4a7a-bfc7-d7343f202585-config-volume\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-config\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151fec2f-192b-4af4-9d4e-10a790100cb9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6tx\" (UniqueName: \"kubernetes.io/projected/151fec2f-192b-4af4-9d4e-10a790100cb9-kube-api-access-rf6tx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114439    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-service-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114480    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5250b4fe-4b08-4a7a-bfc7-d7343f202585-tmp-dir\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.114870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.115551 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk967\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.115551 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-stats-auth\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.115551 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114817    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-tmp\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.115551 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.114968    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.115551 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.113804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-config\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.115890 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.115874    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:05:54.615859476 +0000 UTC m=+30.552579911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:05:54.116195 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.116171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-config\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.116282 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.116214    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.116498 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.116475    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-default-certificate\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.116498 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.116491    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:05:54.116622 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.116502    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.116622 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.116545    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls podName:5250b4fe-4b08-4a7a-bfc7-d7343f202585 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:54.616527843 +0000 UTC m=+30.553248271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls") pod "dns-default-4dcpr" (UID: "5250b4fe-4b08-4a7a-bfc7-d7343f202585") : secret "dns-default-metrics-tls" not found
Oct 30 19:05:54.116622 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.116586    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:05:54.116803 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.116642    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:05:54.616626342 +0000 UTC m=+30.553346787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : secret "router-metrics-certs-default" not found
Oct 30 19:05:54.117586 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.116957    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:05:54.117586 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.117044    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls podName:c0a95351-6d86-4411-903e-3514b69f6bb7 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:54.617029649 +0000 UTC m=+30.553750083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xhlzh" (UID: "c0a95351-6d86-4411-903e-3514b69f6bb7") : secret "samples-operator-tls" not found
Oct 30 19:05:54.117586 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.117176    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.117586 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.117271    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:05:54.117586 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.117286    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd99d66bc-r6hq2: secret "image-registry-tls" not found
Oct 30 19:05:54.117586 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.117326    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls podName:c08977e9-bc18-4071-8748-2c6d7bc94f80 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:54.617313247 +0000 UTC m=+30.554033685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls") pod "image-registry-bd99d66bc-r6hq2" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80") : secret "image-registry-tls" not found
Oct 30 19:05:54.118103 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.117624    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.118103 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.117734    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.118103 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.117878    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5250b4fe-4b08-4a7a-bfc7-d7343f202585-tmp-dir\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.118103 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.117975    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:05:54.118103 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.118018    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert podName:c9e20809-cbf6-491c-b6e9-371655cbd3c6 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:54.618004779 +0000 UTC m=+30.554725257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wqlmz" (UID: "c9e20809-cbf6-491c-b6e9-371655cbd3c6") : secret "networking-console-plugin-cert" not found
Oct 30 19:05:54.118103 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.118091    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.118538 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.118512    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09e9550-86c1-474b-9774-50dd9189f3fa-cert\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:05:54.118659 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.118615    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9e20809-cbf6-491c-b6e9-371655cbd3c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:05:54.118801 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.118777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-stats-auth\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.119199 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.119150    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.119288 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.119196    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-serving-cert\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.119432 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.119408    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151fec2f-192b-4af4-9d4e-10a790100cb9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.119860 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.119839    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-serving-cert\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.120455 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.120437    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.120602 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.120586    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/664daa2d-471e-4380-bfdc-73df3a83ab3e-serving-cert\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.129671 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.129532    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.129671 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.129554    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrvr\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.130130 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.130103    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pfd\" (UniqueName: \"kubernetes.io/projected/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-kube-api-access-55pfd\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.130962 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.130916    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46s4\" (UniqueName: \"kubernetes.io/projected/8c6e8d41-096c-4121-b9c9-a147a7ee010d-kube-api-access-m46s4\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.131674 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.131628    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2b4\" (UniqueName: \"kubernetes.io/projected/1aaa292a-397b-4c16-a897-f4df8a4af32f-kube-api-access-pf2b4\") pod \"network-check-source-6f78d59f99-6d9pz\" (UID: \"1aaa292a-397b-4c16-a897-f4df8a4af32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"
Oct 30 19:05:54.131786 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.131767    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6js6\" (UniqueName: \"kubernetes.io/projected/5250b4fe-4b08-4a7a-bfc7-d7343f202585-kube-api-access-v6js6\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.132435 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.132409    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.133472 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.133445    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hst\" (UniqueName: \"kubernetes.io/projected/6996469c-158c-46af-9e06-a89a9f7137c0-kube-api-access-55hst\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:54.133547 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.133480    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmqs8\" (UniqueName: \"kubernetes.io/projected/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-kube-api-access-kmqs8\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.133547 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.133481    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk78m\" (UniqueName: \"kubernetes.io/projected/a09e9550-86c1-474b-9774-50dd9189f3fa-kube-api-access-xk78m\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:05:54.133893 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.133866    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6tx\" (UniqueName: \"kubernetes.io/projected/151fec2f-192b-4af4-9d4e-10a790100cb9-kube-api-access-rf6tx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.134100 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.134080    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgl6d\" (UniqueName: \"kubernetes.io/projected/664daa2d-471e-4380-bfdc-73df3a83ab3e-kube-api-access-wgl6d\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.134429 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.134407    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmbk\" (UniqueName: \"kubernetes.io/projected/45274ad9-273b-467a-8bf1-27a876267294-kube-api-access-vtmbk\") pod \"volume-data-source-validator-58c9f5f46d-fndkp\" (UID: \"45274ad9-273b-467a-8bf1-27a876267294\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"
Oct 30 19:05:54.135066 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.135045    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk967\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.136130 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.136113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/c0a95351-6d86-4411-903e-3514b69f6bb7-kube-api-access-bb4ct\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:54.148470 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.148449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-98bt4"
Oct 30 19:05:54.162307 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.162286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2xnmc"
Oct 30 19:05:54.174439 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.174421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jkwnj"
Oct 30 19:05:54.185500 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.185472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"
Oct 30 19:05:54.203968 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.203918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:05:54.227725 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.226741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:05:54.234667 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.234408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:05:54.252734 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.252434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:05:54.257158 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.256661    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:54.258614 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.258415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"
Oct 30 19:05:54.276957 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.268570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:05:54.451330 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.451249    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:54.467220 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.467196    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2xnmc"]
Oct 30 19:05:54.471735 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.471712    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-98bt4"]
Oct 30 19:05:54.471875 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:54.471841    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4b6b51_385b_4e07_9eae_5d57cd734d39.slice/crio-58fc931b6298e1b64482a8c903d3d18ea79e704ece34764fe5767a47d0912302 WatchSource:0}: Error finding container 58fc931b6298e1b64482a8c903d3d18ea79e704ece34764fe5767a47d0912302: Status 404 returned error can't find the container with id 58fc931b6298e1b64482a8c903d3d18ea79e704ece34764fe5767a47d0912302
Oct 30 19:05:54.475289 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:54.475266    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9100be_82b2_466c_833c_3ed05703fba2.slice/crio-cbd8e2af697fc4b14f06e3bdf10ba7c5a19c4da2db8fa9a73aefa1831b1f9af6 WatchSource:0}: Error finding container cbd8e2af697fc4b14f06e3bdf10ba7c5a19c4da2db8fa9a73aefa1831b1f9af6: Status 404 returned error can't find the container with id cbd8e2af697fc4b14f06e3bdf10ba7c5a19c4da2db8fa9a73aefa1831b1f9af6
Oct 30 19:05:54.488627 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.488581    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jkwnj"]
Oct 30 19:05:54.495502 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.495386    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"]
Oct 30 19:05:54.503297 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:54.503274    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45274ad9_273b_467a_8bf1_27a876267294.slice/crio-a5b3209e4da4d75788d2c6b68e9ee81bb41741aaad04068e91157cf5869a9c0e WatchSource:0}: Error finding container a5b3209e4da4d75788d2c6b68e9ee81bb41741aaad04068e91157cf5869a9c0e: Status 404 returned error can't find the container with id a5b3209e4da4d75788d2c6b68e9ee81bb41741aaad04068e91157cf5869a9c0e
Oct 30 19:05:54.547437 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.547364    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"]
Oct 30 19:05:54.555519 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:54.552612    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151fec2f_192b_4af4_9d4e_10a790100cb9.slice/crio-b1396fa35e406f4017536ce0642bb361155e0fff2b7ab427af13beb8a5edafb5 WatchSource:0}: Error finding container b1396fa35e406f4017536ce0642bb361155e0fff2b7ab427af13beb8a5edafb5: Status 404 returned error can't find the container with id b1396fa35e406f4017536ce0642bb361155e0fff2b7ab427af13beb8a5edafb5
Oct 30 19:05:54.578944 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.578917    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fskzc"]
Oct 30 19:05:54.583371 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:54.583342    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664daa2d_471e_4380_bfdc_73df3a83ab3e.slice/crio-a84e44f8ce51c5f5b9e483c2dfd6de795a7fee5a68e6880958423432bc5cb292 WatchSource:0}: Error finding container a84e44f8ce51c5f5b9e483c2dfd6de795a7fee5a68e6880958423432bc5cb292: Status 404 returned error can't find the container with id a84e44f8ce51c5f5b9e483c2dfd6de795a7fee5a68e6880958423432bc5cb292
Oct 30 19:05:54.611073 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.611050    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"]
Oct 30 19:05:54.615370 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.615342    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-h6hw8"]
Oct 30 19:05:54.615948 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:54.615923    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c4bb90_1b90_4e9c_9822_5daafa73cdfa.slice/crio-95ad42f668a84230cc02f05dbf573a03b0e75be0300c6a1f7804f4277bc435d8 WatchSource:0}: Error finding container 95ad42f668a84230cc02f05dbf573a03b0e75be0300c6a1f7804f4277bc435d8: Status 404 returned error can't find the container with id 95ad42f668a84230cc02f05dbf573a03b0e75be0300c6a1f7804f4277bc435d8
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.635799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.635865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.635929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.635949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.635967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.635986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.636006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.636024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.635949    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636133    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636160    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls podName:6996469c-158c-46af-9e06-a89a9f7137c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:55.63614066 +0000 UTC m=+31.572861084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rmqbj" (UID: "6996469c-158c-46af-9e06-a89a9f7137c0") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636175    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636179    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:05:55.636168427 +0000 UTC m=+31.572888855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : secret "router-metrics-certs-default" not found
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636082    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636192    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd99d66bc-r6hq2: secret "image-registry-tls" not found
Oct 30 19:05:54.636321 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636198    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls podName:c0a95351-6d86-4411-903e-3514b69f6bb7 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:55.636188194 +0000 UTC m=+31.572908615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xhlzh" (UID: "c0a95351-6d86-4411-903e-3514b69f6bb7") : secret "samples-operator-tls" not found
Oct 30 19:05:54.636835 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636220    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls podName:c08977e9-bc18-4071-8748-2c6d7bc94f80 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:55.636212237 +0000 UTC m=+31.572932662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls") pod "image-registry-bd99d66bc-r6hq2" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80") : secret "image-registry-tls" not found
Oct 30 19:05:54.636835 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636099    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:05:54.636835 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636239    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:05:55.636233217 +0000 UTC m=+31.572953638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:05:54.636835 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636249    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert podName:c9e20809-cbf6-491c-b6e9-371655cbd3c6 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:55.636244293 +0000 UTC m=+31.572964713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wqlmz" (UID: "c9e20809-cbf6-491c-b6e9-371655cbd3c6") : secret "networking-console-plugin-cert" not found
Oct 30 19:05:54.636835 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636270    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:05:54.636835 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636283    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:05:54.636835 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636289    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-775fb78d4b-vms7p: secret "image-registry-tls" not found
Oct 30 19:05:54.636835 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636299    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls podName:5250b4fe-4b08-4a7a-bfc7-d7343f202585 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:55.636289508 +0000 UTC m=+31.573009934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls") pod "dns-default-4dcpr" (UID: "5250b4fe-4b08-4a7a-bfc7-d7343f202585") : secret "dns-default-metrics-tls" not found
Oct 30 19:05:54.636835 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:54.636309    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls podName:8b9d8478-9a05-4d2c-9210-a08764728647 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:55.636304309 +0000 UTC m=+31.573024730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls") pod "image-registry-775fb78d4b-vms7p" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647") : secret "image-registry-tls" not found
Oct 30 19:05:54.651094 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.651039    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nb5x6"]
Oct 30 19:05:54.654805 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:54.654784    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09e9550_86c1_474b_9774_50dd9189f3fa.slice/crio-9adaaeb07f102a51567ae75bb7e4ef9e99112c54befe35f0802e0a538024bdb9 WatchSource:0}: Error finding container 9adaaeb07f102a51567ae75bb7e4ef9e99112c54befe35f0802e0a538024bdb9: Status 404 returned error can't find the container with id 9adaaeb07f102a51567ae75bb7e4ef9e99112c54befe35f0802e0a538024bdb9
Oct 30 19:05:54.673658 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.673633    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"]
Oct 30 19:05:54.676753 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:54.676727    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaa292a_397b_4c16_a897_f4df8a4af32f.slice/crio-ed155d4bb9b9d4727d417f7d28c2c978e960cf563542b52a13b9b0ae958555c3 WatchSource:0}: Error finding container ed155d4bb9b9d4727d417f7d28c2c978e960cf563542b52a13b9b0ae958555c3: Status 404 returned error can't find the container with id ed155d4bb9b9d4727d417f7d28c2c978e960cf563542b52a13b9b0ae958555c3
Oct 30 19:05:54.680142 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.680113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:54.791813 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.791785    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7" event={"ID":"40c4bb90-1b90-4e9c-9822-5daafa73cdfa","Type":"ContainerStarted","Data":"95ad42f668a84230cc02f05dbf573a03b0e75be0300c6a1f7804f4277bc435d8"}
Oct 30 19:05:54.793092 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.793065    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jkwnj" event={"ID":"c91b7f8f-3427-4d04-a5bd-31792aed5075","Type":"ContainerStarted","Data":"96d83477459dbd1076a9cf39c83f149217df748403716803ab79702012ded756"}
Oct 30 19:05:54.793214 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.793104    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jkwnj" event={"ID":"c91b7f8f-3427-4d04-a5bd-31792aed5075","Type":"ContainerStarted","Data":"cd83e00a593d27b08d553990bbab5434c286d2f7fe74a4b21ed5eec63d9dd25d"}
Oct 30 19:05:54.794215 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.794186    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j" event={"ID":"151fec2f-192b-4af4-9d4e-10a790100cb9","Type":"ContainerStarted","Data":"b1396fa35e406f4017536ce0642bb361155e0fff2b7ab427af13beb8a5edafb5"}
Oct 30 19:05:54.795345 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.795323    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp" event={"ID":"45274ad9-273b-467a-8bf1-27a876267294","Type":"ContainerStarted","Data":"a5b3209e4da4d75788d2c6b68e9ee81bb41741aaad04068e91157cf5869a9c0e"}
Oct 30 19:05:54.796501 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.796481    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-h6hw8" event={"ID":"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f","Type":"ContainerStarted","Data":"fd46ba12cd184a7c2e2e90b6afb919b5584f69d7415d56b58adec7462304bb11"}
Oct 30 19:05:54.797675 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.797656    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz" event={"ID":"1aaa292a-397b-4c16-a897-f4df8a4af32f","Type":"ContainerStarted","Data":"ed155d4bb9b9d4727d417f7d28c2c978e960cf563542b52a13b9b0ae958555c3"}
Oct 30 19:05:54.798800 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.798773    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" event={"ID":"664daa2d-471e-4380-bfdc-73df3a83ab3e","Type":"ContainerStarted","Data":"a84e44f8ce51c5f5b9e483c2dfd6de795a7fee5a68e6880958423432bc5cb292"}
Oct 30 19:05:54.800096 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.800070    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2xnmc" event={"ID":"6a4b6b51-385b-4e07-9eae-5d57cd734d39","Type":"ContainerStarted","Data":"b9b0744973136e8ecc987cace20da7799bf0fc322b3d6196af341921e30d7dfd"}
Oct 30 19:05:54.800170 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.800105    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2xnmc" event={"ID":"6a4b6b51-385b-4e07-9eae-5d57cd734d39","Type":"ContainerStarted","Data":"58fc931b6298e1b64482a8c903d3d18ea79e704ece34764fe5767a47d0912302"}
Oct 30 19:05:54.801246 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.801216    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nb5x6" event={"ID":"a09e9550-86c1-474b-9774-50dd9189f3fa","Type":"ContainerStarted","Data":"9adaaeb07f102a51567ae75bb7e4ef9e99112c54befe35f0802e0a538024bdb9"}
Oct 30 19:05:54.802642 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.802623    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-98bt4" event={"ID":"3c9100be-82b2-466c-833c-3ed05703fba2","Type":"ContainerStarted","Data":"cb2f9f390f9120bf803d37f064b8b5716f808437c09e131225cd2c5b17154411"}
Oct 30 19:05:54.802737 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.802656    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-98bt4" event={"ID":"3c9100be-82b2-466c-833c-3ed05703fba2","Type":"ContainerStarted","Data":"cbd8e2af697fc4b14f06e3bdf10ba7c5a19c4da2db8fa9a73aefa1831b1f9af6"}
Oct 30 19:05:54.812121 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.812082    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jkwnj" podStartSLOduration=373.812071753 podStartE2EDuration="6m13.812071753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:54.810549646 +0000 UTC m=+30.747270089" watchObservedRunningTime="2026-10-30 19:05:54.812071753 +0000 UTC m=+30.748792195"
Oct 30 19:05:54.826607 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.826574    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-98bt4" podStartSLOduration=373.826563875 podStartE2EDuration="6m13.826563875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:54.826308193 +0000 UTC m=+30.763028637" watchObservedRunningTime="2026-10-30 19:05:54.826563875 +0000 UTC m=+30.763284314"
Oct 30 19:05:54.842652 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:54.842616    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2xnmc" podStartSLOduration=373.842605914 podStartE2EDuration="6m13.842605914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:54.841550594 +0000 UTC m=+30.778271038" watchObservedRunningTime="2026-10-30 19:05:54.842605914 +0000 UTC m=+30.779326356"
Oct 30 19:05:55.552495 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.550498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:55.552495 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.551053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:55.552495 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.551534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:55.555702 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.554926    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:05:55.555702 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.555376    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-89jvk\""
Oct 30 19:05:55.555702 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.555491    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fljvz\""
Oct 30 19:05:55.555702 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.555531    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tc7v2\""
Oct 30 19:05:55.556857 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.556740    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:05:55.647657 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.647622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:55.647899 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.647678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:55.647899 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.647736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:55.647899 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.647770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:55.647899 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.647810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:55.647899 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.647840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:05:55.647899 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.647885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:55.648197 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:55.647939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:55.648197 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.648097    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:05:55.648197 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.648110    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-775fb78d4b-vms7p: secret "image-registry-tls" not found
Oct 30 19:05:55.648197 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.648167    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls podName:8b9d8478-9a05-4d2c-9210-a08764728647 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:57.648148274 +0000 UTC m=+33.584868700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls") pod "image-registry-775fb78d4b-vms7p" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647") : secret "image-registry-tls" not found
Oct 30 19:05:55.648727 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.648708    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:05:55.648815 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.648769    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls podName:5250b4fe-4b08-4a7a-bfc7-d7343f202585 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:57.64875238 +0000 UTC m=+33.585472805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls") pod "dns-default-4dcpr" (UID: "5250b4fe-4b08-4a7a-bfc7-d7343f202585") : secret "dns-default-metrics-tls" not found
Oct 30 19:05:55.648924 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.648911    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:05:55.648973 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.648954    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls podName:c0a95351-6d86-4411-903e-3514b69f6bb7 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:57.648942381 +0000 UTC m=+33.585662816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xhlzh" (UID: "c0a95351-6d86-4411-903e-3514b69f6bb7") : secret "samples-operator-tls" not found
Oct 30 19:05:55.649071 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649018    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:05:55.649071 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649029    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd99d66bc-r6hq2: secret "image-registry-tls" not found
Oct 30 19:05:55.649071 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649061    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls podName:c08977e9-bc18-4071-8748-2c6d7bc94f80 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:57.649050659 +0000 UTC m=+33.585771096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls") pod "image-registry-bd99d66bc-r6hq2" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80") : secret "image-registry-tls" not found
Oct 30 19:05:55.649231 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649117    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:05:55.649231 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649149    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:05:57.649138182 +0000 UTC m=+33.585858618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : secret "router-metrics-certs-default" not found
Oct 30 19:05:55.649231 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649214    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:05:57.64920386 +0000 UTC m=+33.585924299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:05:55.649458 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649269    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:05:55.649458 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649301    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert podName:c9e20809-cbf6-491c-b6e9-371655cbd3c6 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:57.649291106 +0000 UTC m=+33.586011542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wqlmz" (UID: "c9e20809-cbf6-491c-b6e9-371655cbd3c6") : secret "networking-console-plugin-cert" not found
Oct 30 19:05:55.649458 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649354    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:05:55.649458 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:55.649388    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls podName:6996469c-158c-46af-9e06-a89a9f7137c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:57.649377421 +0000 UTC m=+33.586097857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rmqbj" (UID: "6996469c-158c-46af-9e06-a89a9f7137c0") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:05:57.485308 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.485273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:57.641744 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.641639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.670339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.670392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.670432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.670503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.670601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.670637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.670667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.670756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.670939    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671001    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.670982443 +0000 UTC m=+37.607702878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : secret "router-metrics-certs-default" not found
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671407    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.671392071 +0000 UTC m=+37.608112508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671475    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671508    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert podName:c9e20809-cbf6-491c-b6e9-371655cbd3c6 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.671497605 +0000 UTC m=+37.608218040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wqlmz" (UID: "c9e20809-cbf6-491c-b6e9-371655cbd3c6") : secret "networking-console-plugin-cert" not found
Oct 30 19:05:57.672013 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671558    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671590    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls podName:6996469c-158c-46af-9e06-a89a9f7137c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.671578399 +0000 UTC m=+37.608298833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rmqbj" (UID: "6996469c-158c-46af-9e06-a89a9f7137c0") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671661    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671671    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-775fb78d4b-vms7p: secret "image-registry-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671719    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls podName:8b9d8478-9a05-4d2c-9210-a08764728647 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.671708007 +0000 UTC m=+37.608428433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls") pod "image-registry-775fb78d4b-vms7p" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647") : secret "image-registry-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671773    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671805    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls podName:5250b4fe-4b08-4a7a-bfc7-d7343f202585 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.671794164 +0000 UTC m=+37.608514600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls") pod "dns-default-4dcpr" (UID: "5250b4fe-4b08-4a7a-bfc7-d7343f202585") : secret "dns-default-metrics-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671855    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671887    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls podName:c0a95351-6d86-4411-903e-3514b69f6bb7 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.671877562 +0000 UTC m=+37.608597997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xhlzh" (UID: "c0a95351-6d86-4411-903e-3514b69f6bb7") : secret "samples-operator-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671940    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671950    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd99d66bc-r6hq2: secret "image-registry-tls" not found
Oct 30 19:05:57.672819 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:57.671978    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls podName:c08977e9-bc18-4071-8748-2c6d7bc94f80 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.671968145 +0000 UTC m=+37.608688571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls") pod "image-registry-bd99d66bc-r6hq2" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80") : secret "image-registry-tls" not found
Oct 30 19:05:57.825841 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:57.825806    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp" event={"ID":"45274ad9-273b-467a-8bf1-27a876267294","Type":"ContainerStarted","Data":"9e96fa35d1fd574cf45cf0aad7fe260012dc230951f0d427c3cc471a42f44ab6"}
Oct 30 19:05:58.027727 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:58.027576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:58.197154 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:58.197117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:58.275487 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:58.275447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:05:58.275818 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:58.275779    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:05:58.275912 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:05:58.275845    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:30.275825795 +0000 UTC m=+66.212546220 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : secret "metrics-daemon-secret" not found
Oct 30 19:05:58.377234 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:58.377156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:58.377383 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:58.377246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:58.380249 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:58.380227    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:58.382308 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:58.382264    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:58.589749 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:58.589717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:05:58.605825 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:58.605797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:05:59.047401 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.047322    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp" podStartSLOduration=343.870398629 podStartE2EDuration="5m46.04727994s" totalImagesPullingTime="2.176881311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:57.846384948 +0000 UTC m=+33.783105392" watchObservedRunningTime="2026-10-30 19:05:59.04727994 +0000 UTC m=+34.984000384"
Oct 30 19:05:59.048347 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.048323    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kj2xn"]
Oct 30 19:05:59.054572 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.054551    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9s7mb"]
Oct 30 19:05:59.228450 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:59.228366    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797c4f9e_bf72_41af_a1df_f39270f68e1a.slice/crio-1f4e84a81880dae7b48902ade6f1ec5a4d2708bc84b8755086f6cd3ea282ac49 WatchSource:0}: Error finding container 1f4e84a81880dae7b48902ade6f1ec5a4d2708bc84b8755086f6cd3ea282ac49: Status 404 returned error can't find the container with id 1f4e84a81880dae7b48902ade6f1ec5a4d2708bc84b8755086f6cd3ea282ac49
Oct 30 19:05:59.229475 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:05:59.229444    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbe6234_bffe_4a9f_a543_8c509b08357e.slice/crio-4722938f7e9c1ab3faeb57149c44afe1dac61833d8c5ecd2c8e3b400db486889 WatchSource:0}: Error finding container 4722938f7e9c1ab3faeb57149c44afe1dac61833d8c5ecd2c8e3b400db486889: Status 404 returned error can't find the container with id 4722938f7e9c1ab3faeb57149c44afe1dac61833d8c5ecd2c8e3b400db486889
Oct 30 19:05:59.414555 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.414530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:59.583878 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.583812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:59.833019 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.831626    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x4lg8"]
Oct 30 19:05:59.853033 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.852300    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/0.log"
Oct 30 19:05:59.853033 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.852342    2577 generic.go:356] "Generic (PLEG): container finished" podID="664daa2d-471e-4380-bfdc-73df3a83ab3e" containerID="2bcba1d4d3f92bc6b661497681a7c5218f746c455863b6d83b97bcd2ff838572" exitCode=255
Oct 30 19:05:59.853033 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.852657    2577 scope.go:122] "RemoveContainer" containerID="2bcba1d4d3f92bc6b661497681a7c5218f746c455863b6d83b97bcd2ff838572"
Oct 30 19:05:59.860710 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.860615    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4lg8"]
Oct 30 19:05:59.860710 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.860662    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7" event={"ID":"40c4bb90-1b90-4e9c-9822-5daafa73cdfa","Type":"ContainerStarted","Data":"9d3f51aea9a11ec6df5cc6fb0defd685ffabf9ac37ee993b34faca222a63d3d4"}
Oct 30 19:05:59.861131 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.860757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:05:59.861627 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.861208    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j" event={"ID":"151fec2f-192b-4af4-9d4e-10a790100cb9","Type":"ContainerStarted","Data":"b3f2e00bcbcb949c9feafd22168cc3abce728a37aa0bec6383e883aaee46fcc2"}
Oct 30 19:05:59.861627 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.861245    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-h6hw8" event={"ID":"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f","Type":"ContainerStarted","Data":"28732040500bcbecd4c6ba544d087b5f2b4351809934d0ad4b78ab0a87440952"}
Oct 30 19:05:59.861627 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.861261    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" event={"ID":"664daa2d-471e-4380-bfdc-73df3a83ab3e","Type":"ContainerDied","Data":"2bcba1d4d3f92bc6b661497681a7c5218f746c455863b6d83b97bcd2ff838572"}
Oct 30 19:05:59.861627 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.861276    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nb5x6" event={"ID":"a09e9550-86c1-474b-9774-50dd9189f3fa","Type":"ContainerStarted","Data":"de2cd9436500c49abce460ef5b1a85d476b9433b18934dd5f78e104194fedff9"}
Oct 30 19:05:59.861627 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.861290    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kj2xn" event={"ID":"797c4f9e-bf72-41af-a1df-f39270f68e1a","Type":"ContainerStarted","Data":"1f4e84a81880dae7b48902ade6f1ec5a4d2708bc84b8755086f6cd3ea282ac49"}
Oct 30 19:05:59.865164 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.865032    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9s7mb" event={"ID":"3cbe6234-bffe-4a9f-a543-8c509b08357e","Type":"ContainerStarted","Data":"4722938f7e9c1ab3faeb57149c44afe1dac61833d8c5ecd2c8e3b400db486889"}
Oct 30 19:05:59.866009 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.865376    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:05:59.866379 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.866358    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:05:59.866716 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.866485    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fsc2n\""
Oct 30 19:05:59.869097 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.867036    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7" podStartSLOduration=338.172468827 podStartE2EDuration="5m42.867024814s" totalImagesPullingTime="4.694555987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:59.866118185 +0000 UTC m=+35.802838629" watchObservedRunningTime="2026-10-30 19:05:59.867024814 +0000 UTC m=+35.803745257"
Oct 30 19:05:59.899762 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.898557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-data-volume\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:05:59.899762 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.898616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxcpn\" (UniqueName: \"kubernetes.io/projected/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-api-access-hxcpn\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:05:59.899762 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.898655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:05:59.899762 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.898739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-crio-socket\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:05:59.899762 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.898902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:05:59.908722 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.908015    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-h6hw8" podStartSLOduration=347.216923891 podStartE2EDuration="5m51.908000556s" totalImagesPullingTime="4.691076665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:59.90718786 +0000 UTC m=+35.843908302" watchObservedRunningTime="2026-10-30 19:05:59.908000556 +0000 UTC m=+35.844720989"
Oct 30 19:05:59.929729 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.925766    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nb5x6" podStartSLOduration=1.877530215 podStartE2EDuration="6.925745289s" totalImagesPullingTime="5.048215074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:59.924377836 +0000 UTC m=+35.861098279" watchObservedRunningTime="2026-10-30 19:05:59.925745289 +0000 UTC m=+35.862465733"
Oct 30 19:05:59.950614 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.938305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:59.978719 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.972294    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j" podStartSLOduration=344.22267983 podStartE2EDuration="5m48.972274977s" totalImagesPullingTime="4.749595147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:59.970882142 +0000 UTC m=+35.907602586" watchObservedRunningTime="2026-10-30 19:05:59.972274977 +0000 UTC m=+35.908995425"
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:05:59.999832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-data-volume\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.000232    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-data-volume\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.000289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxcpn\" (UniqueName: \"kubernetes.io/projected/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-api-access-hxcpn\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.000350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.000390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-crio-socket\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.000482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:00.000637    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:00.000712    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:00.500675216 +0000 UTC m=+36.437395663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : configmap "kube-rbac-proxy" not found
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:00.001190    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:00.001243    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:00.501227231 +0000 UTC m=+36.437947691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:00.001384 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.001298    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-crio-socket\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.024257 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.024190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxcpn\" (UniqueName: \"kubernetes.io/projected/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-api-access-hxcpn\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.111230 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.111166    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:00.366441 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.366345    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"]
Oct 30 19:06:00.380523 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.380495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"
Oct 30 19:06:00.381749 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.380359    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"]
Oct 30 19:06:00.386431 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.386406    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:00.386562 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.386468    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:06:00.386562 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.386514    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6qlhx\""
Oct 30 19:06:00.404914 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.404886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/ffc0076c-0b02-421c-851a-616ecf70e075-kube-api-access-b5htb\") pod \"migrator-66df5d496b-lqgxj\" (UID: \"ffc0076c-0b02-421c-851a-616ecf70e075\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"
Oct 30 19:06:00.442089 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.442017    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:00.506726 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.506257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/ffc0076c-0b02-421c-851a-616ecf70e075-kube-api-access-b5htb\") pod \"migrator-66df5d496b-lqgxj\" (UID: \"ffc0076c-0b02-421c-851a-616ecf70e075\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"
Oct 30 19:06:00.506726 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.506656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.506943 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.506797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:00.506992 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:00.506968    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:06:00.507037 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:00.507029    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.507009686 +0000 UTC m=+37.443730128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : configmap "kube-rbac-proxy" not found
Oct 30 19:06:00.507252 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:00.507229    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:00.507364 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:00.507287    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:01.507272862 +0000 UTC m=+37.443993286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:00.519810 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.519783    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/ffc0076c-0b02-421c-851a-616ecf70e075-kube-api-access-b5htb\") pod \"migrator-66df5d496b-lqgxj\" (UID: \"ffc0076c-0b02-421c-851a-616ecf70e075\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"
Oct 30 19:06:00.624293 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.624216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:00.716478 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.716284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"
Oct 30 19:06:00.788569 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.788532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:00.873010 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.872572    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz" event={"ID":"1aaa292a-397b-4c16-a897-f4df8a4af32f","Type":"ContainerStarted","Data":"ad902db1d33921b0ed66e6cb1399e23e4a094a91fd8d9546a830d8aac0a6544f"}
Oct 30 19:06:00.874557 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.874492    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/1.log"
Oct 30 19:06:00.875157 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.875137    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/0.log"
Oct 30 19:06:00.875241 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.875179    2577 generic.go:356] "Generic (PLEG): container finished" podID="664daa2d-471e-4380-bfdc-73df3a83ab3e" containerID="1e3532dddac09b5586753aa2a69e1243fc6678c9ea895299412cdbb3a073465b" exitCode=255
Oct 30 19:06:00.875293 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.875267    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" event={"ID":"664daa2d-471e-4380-bfdc-73df3a83ab3e","Type":"ContainerDied","Data":"1e3532dddac09b5586753aa2a69e1243fc6678c9ea895299412cdbb3a073465b"}
Oct 30 19:06:00.875348 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.875330    2577 scope.go:122] "RemoveContainer" containerID="2bcba1d4d3f92bc6b661497681a7c5218f746c455863b6d83b97bcd2ff838572"
Oct 30 19:06:00.875536 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.875518    2577 scope.go:122] "RemoveContainer" containerID="1e3532dddac09b5586753aa2a69e1243fc6678c9ea895299412cdbb3a073465b"
Oct 30 19:06:00.875784 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:00.875758    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fskzc_openshift-console-operator(664daa2d-471e-4380-bfdc-73df3a83ab3e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" podUID="664daa2d-471e-4380-bfdc-73df3a83ab3e"
Oct 30 19:06:00.894948 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:00.894897    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz" podStartSLOduration=338.94165314 podStartE2EDuration="5m44.89488159s" totalImagesPullingTime="5.95322845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:00.894349244 +0000 UTC m=+36.831069687" watchObservedRunningTime="2026-10-30 19:06:00.89488159 +0000 UTC m=+36.831602036"
Oct 30 19:06:01.033309 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.033276    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"]
Oct 30 19:06:01.036801 ip-10-0-130-164 kubenswrapper[2577]: W1030 19:06:01.036774    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc0076c_0b02_421c_851a_616ecf70e075.slice/crio-53d5a4c69bd68265e94eadf485e8ca53248bf9c67a94fda25844f2e50f0fce13 WatchSource:0}: Error finding container 53d5a4c69bd68265e94eadf485e8ca53248bf9c67a94fda25844f2e50f0fce13: Status 404 returned error can't find the container with id 53d5a4c69bd68265e94eadf485e8ca53248bf9c67a94fda25844f2e50f0fce13
Oct 30 19:06:01.302707 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.302416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:01.463870 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.463835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:01.525831 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.525431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:01.525831 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.525547    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:06:01.525831 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.525633    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:03.525612086 +0000 UTC m=+39.462332521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : configmap "kube-rbac-proxy" not found
Oct 30 19:06:01.525831 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.525642    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:01.525831 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.525729    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:03.5257121 +0000 UTC m=+39.462432524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:01.525831 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.525547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:01.727026 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.726990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:01.727208 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.727054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:01.727208 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.727140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:01.727208 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.727173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:01.727335 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.727210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:01.727335 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727220    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:01.727335 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.727240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:01.727335 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727256    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:01.727335 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727266    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:06:01.727335 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727276    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-775fb78d4b-vms7p: secret "image-registry-tls" not found
Oct 30 19:06:01.727335 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727286    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls podName:6996469c-158c-46af-9e06-a89a9f7137c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.727268246 +0000 UTC m=+45.663988670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rmqbj" (UID: "6996469c-158c-46af-9e06-a89a9f7137c0") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:01.727335 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727321    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls podName:8b9d8478-9a05-4d2c-9210-a08764728647 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.727308382 +0000 UTC m=+45.664028803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls") pod "image-registry-775fb78d4b-vms7p" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647") : secret "image-registry-tls" not found
Oct 30 19:06:01.727335 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727334    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls podName:5250b4fe-4b08-4a7a-bfc7-d7343f202585 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.727328369 +0000 UTC m=+45.664048790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls") pod "dns-default-4dcpr" (UID: "5250b4fe-4b08-4a7a-bfc7-d7343f202585") : secret "dns-default-metrics-tls" not found
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727344    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.727339303 +0000 UTC m=+45.664059723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.727272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.727376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727383    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727416    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.727404907 +0000 UTC m=+45.664125349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : secret "router-metrics-certs-default" not found
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727438    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727461    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert podName:c9e20809-cbf6-491c-b6e9-371655cbd3c6 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.727454922 +0000 UTC m=+45.664175343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wqlmz" (UID: "c9e20809-cbf6-491c-b6e9-371655cbd3c6") : secret "networking-console-plugin-cert" not found
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727474    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727473    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727509    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd99d66bc-r6hq2: secret "image-registry-tls" not found
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727513    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls podName:c0a95351-6d86-4411-903e-3514b69f6bb7 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.72750271 +0000 UTC m=+45.664223147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xhlzh" (UID: "c0a95351-6d86-4411-903e-3514b69f6bb7") : secret "samples-operator-tls" not found
Oct 30 19:06:01.727643 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.727551    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls podName:c08977e9-bc18-4071-8748-2c6d7bc94f80 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.727534965 +0000 UTC m=+45.664255387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls") pod "image-registry-bd99d66bc-r6hq2" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80") : secret "image-registry-tls" not found
Oct 30 19:06:01.881276 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.881244    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/1.log"
Oct 30 19:06:01.881780 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.881639    2577 scope.go:122] "RemoveContainer" containerID="1e3532dddac09b5586753aa2a69e1243fc6678c9ea895299412cdbb3a073465b"
Oct 30 19:06:01.881892 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:01.881870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fskzc_openshift-console-operator(664daa2d-471e-4380-bfdc-73df3a83ab3e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" podUID="664daa2d-471e-4380-bfdc-73df3a83ab3e"
Oct 30 19:06:01.884074 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:01.884046    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj" event={"ID":"ffc0076c-0b02-421c-851a-616ecf70e075","Type":"ContainerStarted","Data":"53d5a4c69bd68265e94eadf485e8ca53248bf9c67a94fda25844f2e50f0fce13"}
Oct 30 19:06:02.354061 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:02.353982    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:02.524461 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:02.524432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:02.889564 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:02.889468    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9s7mb" event={"ID":"3cbe6234-bffe-4a9f-a543-8c509b08357e","Type":"ContainerStarted","Data":"7b74afb31253d1f41d3ffe9d10b0d6850d7693f0cc270e4a5466147785569803"}
Oct 30 19:06:02.890024 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:02.889599    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:06:02.891562 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:02.891537    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj" event={"ID":"ffc0076c-0b02-421c-851a-616ecf70e075","Type":"ContainerStarted","Data":"2feb26f9e92f9b5d86c908b2518f67810b753c2c75ea0b07f529a754d1800803"}
Oct 30 19:06:02.907637 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:02.907592    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9s7mb" podStartSLOduration=36.130398329 podStartE2EDuration="38.907580048s" totalImagesPullingTime="2.777181719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:02.906195349 +0000 UTC m=+38.842915804" watchObservedRunningTime="2026-10-30 19:06:02.907580048 +0000 UTC m=+38.844300490"
Oct 30 19:06:02.924018 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:02.923971    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj" podStartSLOduration=1.376803636 podStartE2EDuration="2.923957225s" totalImagesPullingTime="1.547153589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:02.922100604 +0000 UTC m=+38.858821047" watchObservedRunningTime="2026-10-30 19:06:02.923957225 +0000 UTC m=+38.860677700"
Oct 30 19:06:03.544617 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:03.544582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:03.544805 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:03.544672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:03.544805 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:03.544779    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:06:03.544805 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:03.544779    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:03.544942 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:03.544844    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.544829478 +0000 UTC m=+43.481549898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : configmap "kube-rbac-proxy" not found
Oct 30 19:06:03.544942 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:03.544863    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.544852573 +0000 UTC m=+43.481572996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:04.228760 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:04.228719    2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:04.229238 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:04.228793    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:04.229238 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:04.229224    2577 scope.go:122] "RemoveContainer" containerID="1e3532dddac09b5586753aa2a69e1243fc6678c9ea895299412cdbb3a073465b"
Oct 30 19:06:04.229466 ip-10-0-130-164 kubenswrapper[2577]: E1030 19:06:04.229432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fskzc_openshift-console-operator(664daa2d-471e-4380-bfdc-73df3a83ab3e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" podUID="664daa2d-471e-4380-bfdc-73df3a83ab3e"
Oct 30 19:06:04.899740 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:04.899703    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kj2xn" event={"ID":"797c4f9e-bf72-41af-a1df-f39270f68e1a","Type":"ContainerStarted","Data":"ffe4d536ea713d4bd9dbd33f5588ec815ce2b34d7ba6dc7f02e41e452f776200"}
Oct 30 19:06:04.915979 ip-10-0-130-164 kubenswrapper[2577]: I1030 19:06:04.915926    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kj2xn" podStartSLOduration=34.449077265 podStartE2EDuration="39.915909824s" totalImagesPullingTime="5.466832559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:04.915188646 +0000 UTC m=+40.851909091" watchObservedRunningTime="2026-10-30 19:06:04.915909824 +0000 UTC m=+40.852630270"
Oct 30 19:06:05.037292 ip-10-0-130-164 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:06:05.054147 ip-10-0-130-164 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:06:05.054382 ip-10-0-130-164 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:06:05.054440 ip-10-0-130-164 systemd[1]: kubelet.service: Consumed 2.991s CPU time.
Oct 30 19:06:05.066154 ip-10-0-130-164 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:06:05.172919 ip-10-0-130-164 kubenswrapper[4925]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:06:05.172919 ip-10-0-130-164 kubenswrapper[4925]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:06:05.172919 ip-10-0-130-164 kubenswrapper[4925]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:06:05.172919 ip-10-0-130-164 kubenswrapper[4925]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:06:05.175037 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.174945    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:05.175037 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175037    4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175041    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175044    4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175046    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175049    4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175052    4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175054    4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175057    4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175060    4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175063    4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175065    4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175068    4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175070    4925 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175073    4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175076    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175078    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175081    4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175085    4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175089    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:05.175111 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175092    4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175095    4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175098    4925 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175101    4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175104    4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175113    4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175117    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175120    4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175123    4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175126    4925 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175128    4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175131    4925 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175133    4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175136    4925 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175139    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175141    4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175144    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175146    4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175149    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175151    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:05.175592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175154    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175156    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175159    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175162    4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175164    4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175167    4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175169    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175172    4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175174    4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175176    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175180    4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175182    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175185    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175187    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175190    4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175192    4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175195    4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175197    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175200    4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:05.176180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175203    4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175207    4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175210    4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175213    4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175215    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175218    4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175220    4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175223    4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175226    4925 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175228    4925 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175231    4925 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175233    4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175236    4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175239    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175242    4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175244    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175247    4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175249    4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175252    4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:05.176722 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175254    4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175256    4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175259    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175262    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175265    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175268    4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175270    4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175273    4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175276    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175278    4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175282    4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175285    4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175287    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175290    4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175292    4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175294    4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175297    4925 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175299    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175302    4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:05.177199 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175304    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175307    4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175309    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175311    4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175314    4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175317    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175320    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175322    4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.175326    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176095    4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176102    4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176105    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176109    4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176111    4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176114    4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176117    4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176119    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176122    4925 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176126    4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:05.177703 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176129    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176132    4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176134    4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176137    4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176140    4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176142    4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176145    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176147    4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176150    4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176153    4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176155    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176158    4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176161    4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176164    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176166    4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176169    4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176172    4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176174    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176177    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:05.178200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176180    4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176183    4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176185    4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176189    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176192    4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176195    4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176197    4925 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176200    4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176202    4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176205    4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176207    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176210    4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176212    4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176214    4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176217    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176220    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176223    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176229    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176232    4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176234    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:05.178664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176237    4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176239    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176242    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176246    4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176251    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176254    4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176257    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176259    4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176262    4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176265    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176267    4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176270    4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176272    4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176275    4925 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176277    4925 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176280    4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176285    4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176288    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176290    4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:05.179175 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176293    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176296    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176298    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176300    4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176303    4925 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176306    4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176308    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176312    4925 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176315    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176317    4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176321    4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176324    4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176326    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176329    4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176331    4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176334    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176336    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176339    4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176341    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176344    4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:05.179647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176346    4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176349    4925 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176352    4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176354    4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176356    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176359    4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176361    4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176364    4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176366    4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176369    4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176372    4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176374    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176377    4925 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176379    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176382    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176384    4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176386    4925 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176389    4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176944    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176953    4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:05.180157 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176956    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176958    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176962    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176965    4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176967    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176970    4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176972    4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176975    4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176978    4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176980    4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176983    4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176987    4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176991    4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176994    4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.176997    4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177000    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177003    4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177005    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177008    4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:05.180647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177010    4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177013    4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177015    4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177024    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177027    4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177030    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177032    4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177035    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177037    4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177040    4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177043    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177045    4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177049    4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177052    4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177054    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177057    4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177060    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177062    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177064    4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:05.181167 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177067    4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177069    4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177072    4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177074    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177077    4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177080    4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177082    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177085    4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177087    4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177090    4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177092    4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177095    4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177097    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177100    4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177102    4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177104    4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177107    4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177115    4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177117    4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:05.181640 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177120    4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177122    4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177125    4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177128    4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177132    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177135    4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177140    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177142    4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177145    4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177147    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177151    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177153    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177156    4925 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177159    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177161    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177163    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177166    4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177168    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177171    4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:05.182125 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177173    4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177175    4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177178    4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177180    4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177183    4925 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177185    4925 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177188    4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177192    4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177195    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177197    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177200    4925 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177202    4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177212    4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177215    4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177217    4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177220    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177222    4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177225    4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177227    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177230    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:05.182642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177233    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177236    4925 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177239    4925 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177241    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177243    4925 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177246    4925 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177248    4925 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.177251    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177347    4925 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177354    4925 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177370    4925 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177374    4925 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177379    4925 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177382    4925 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177386    4925 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177390    4925 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177393    4925 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177397    4925 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177400    4925 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177405    4925 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177408    4925 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177411    4925 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177413    4925 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:06:05.183163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177416    4925 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177419    4925 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177428    4925 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177434    4925 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177437    4925 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177440    4925 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177444    4925 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177447    4925 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177451    4925 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177454    4925 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177457    4925 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177461    4925 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177464    4925 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177466    4925 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177469    4925 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177472    4925 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177475    4925 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177480    4925 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177482    4925 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177485    4925 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177488    4925 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177491    4925 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177494    4925 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177500    4925 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177503    4925 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:06:05.183747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177506    4925 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177509    4925 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177512    4925 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177518    4925 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177522    4925 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177525    4925 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177527    4925 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177530    4925 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177533    4925 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177536    4925 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177544    4925 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177547    4925 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177550    4925 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177553    4925 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177558    4925 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177561    4925 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177564    4925 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177567    4925 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177570    4925 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177573    4925 flags.go:64] FLAG: --help="false"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177576    4925 flags.go:64] FLAG: --hostname-override="ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177579    4925 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177582    4925 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:06:05.184851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177585    4925 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177588    4925 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177592    4925 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177594    4925 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177597    4925 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177600    4925 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177603    4925 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177606    4925 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177609    4925 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177611    4925 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177614    4925 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177617    4925 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177621    4925 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177624    4925 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177627    4925 flags.go:64] FLAG: --lock-file=""
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177629    4925 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177632    4925 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177635    4925 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177641    4925 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177644    4925 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177652    4925 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177655    4925 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177658    4925 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177662    4925 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:06:05.185438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177665    4925 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177668    4925 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177672    4925 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177675    4925 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177679    4925 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177698    4925 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177703    4925 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177706    4925 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177709    4925 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177712    4925 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177715    4925 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177718    4925 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177725    4925 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177728    4925 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177731    4925 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177734    4925 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177737    4925 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177739    4925 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177742    4925 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177745    4925 flags.go:64] FLAG: --port="10250"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177749    4925 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177752    4925 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02c0239956684b658"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177755    4925 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177759    4925 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177762    4925 flags.go:64] FLAG: --register-node="true"
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177765    4925 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:06:05.186052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177769    4925 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177772    4925 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177775    4925 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177784    4925 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177789    4925 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177792    4925 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177795    4925 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177798    4925 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177800    4925 flags.go:64] FLAG: --runonce="false"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177803    4925 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177806    4925 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177809    4925 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177812    4925 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177815    4925 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177818    4925 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177821    4925 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177824    4925 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177827    4925 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177829    4925 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177832    4925 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177835    4925 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177838    4925 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177840    4925 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177843    4925 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177848    4925 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:06:05.186739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177851    4925 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177855    4925 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177862    4925 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177864    4925 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177867    4925 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177870    4925 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177873    4925 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177876    4925 flags.go:64] FLAG: --v="2"
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177880    4925 flags.go:64] FLAG: --version="false"
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177884    4925 flags.go:64] FLAG: --vmodule=""
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177888    4925 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.177900    4925 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178082    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178086    4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178089    4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178091    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178094    4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178096    4925 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178099    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178102    4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178106    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:05.187354 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178109    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178113    4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178117    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178119    4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178123    4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178126    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178128    4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178131    4925 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178133    4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178135    4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178138    4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178141    4925 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178144    4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178147    4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178150    4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178152    4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178155    4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178157    4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178160    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:05.187883 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178162    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178177    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178180    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178183    4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178192    4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178195    4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178198    4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178201    4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178204    4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178206    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178209    4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178212    4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178214    4925 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178217    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178220    4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178222    4925 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178224    4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178227    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178230    4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178232    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:05.188396 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178234    4925 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178237    4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178240    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178242    4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178245    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178248    4925 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178251    4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178254    4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178256    4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178259    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178261    4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178264    4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178272    4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178275    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178277    4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178282    4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178284    4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178292    4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178295    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:05.189023 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178298    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178300    4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178302    4925 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178305    4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178307    4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178310    4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178312    4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178315    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178318    4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178320    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178323    4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178326    4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178329    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178331    4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178333    4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178336    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178338    4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178340    4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178344    4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178347    4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:05.189677 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178349    4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178352    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178354    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178357    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178359    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178362    4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178364    4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178367    4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178370    4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178373    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178375    4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178384    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178387    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178390    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178392    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178394    4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178397    4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178399    4925 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:05.191038 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178402    4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.178427    4925 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178474    4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178479    4925 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178482    4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178485    4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178487    4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178490    4925 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178493    4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178495    4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178498    4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178500    4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178503    4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178507    4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178509    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178512    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:05.191523 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178515    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178517    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178519    4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178522    4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178524    4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178527    4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178531    4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178533    4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178536    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178538    4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178546    4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178549    4925 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178552    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178554    4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178560    4925 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178563    4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178565    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178567    4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178570    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178572    4925 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:05.192035 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178575    4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178578    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178580    4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178583    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178585    4925 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178588    4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178590    4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178592    4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178595    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178598    4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178601    4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178603    4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178605    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178608    4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178610    4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178613    4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178616    4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178618    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178622    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:05.192556 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178625    4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178627    4925 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178630    4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178632    4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178640    4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178643    4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178645    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178648    4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178650    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178653    4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178656    4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178659    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178661    4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178664    4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178666    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178669    4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178671    4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178674    4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178677    4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178679    4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:05.193063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178682    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178701    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178710    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178713    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178715    4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178718    4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178720    4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178723    4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178725    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178728    4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178730    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178734    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178736    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178739    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178741    4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178744    4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178746    4925 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178755    4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178759    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178761    4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:05.193550 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178764    4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178766    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178769    4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178771    4925 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178773    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178777    4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178781    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178784    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178788    4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178791    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178794    4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178796    4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.178800    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.182730    4925 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.182744    4925 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.182765    4925 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.182770    4925 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182827    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:05.194059 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182832    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182835    4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182838    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182841    4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182843    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182846    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182848    4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182851    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182853    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182856    4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182859    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182862    4925 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182865    4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182868    4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182870    4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182873    4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182875    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182878    4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182880    4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:05.194519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182883    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182886    4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182888    4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182891    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182894    4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182896    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182899    4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182903    4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182905    4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182908    4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182911    4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182915    4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182918    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182922    4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182925    4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182927    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182930    4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182932    4925 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182935    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:05.194993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182937    4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182940    4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182942    4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182945    4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182947    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182950    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182953    4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182955    4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182958    4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182961    4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182963    4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182966    4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182968    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182971    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182973    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182976    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182978    4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182981    4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182983    4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:05.195456 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182987    4925 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182989    4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182992    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182994    4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182997    4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.182999    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183002    4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183004    4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183007    4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183009    4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183012    4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183014    4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183016    4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183019    4925 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183022    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183025    4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183027    4925 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183030    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183032    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183035    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:05.195922 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183037    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183040    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183044    4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183047    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183049    4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183052    4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183054    4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183057    4925 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183059    4925 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183062    4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183064    4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183066    4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183069    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183072    4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183074    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183077    4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183079    4925 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183081    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183084    4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:05.196434 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183086    4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183089    4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183092    4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183094    4925 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183096    4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183099    4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183101    4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183103    4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183106    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.183132    4925 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183276    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183280    4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183283    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183286    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183289    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183292    4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:05.197041 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183295    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183297    4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183299    4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183302    4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183304    4925 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183307    4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183309    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183311    4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183314    4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183317    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183320    4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183322    4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183324    4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183327    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183329    4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183331    4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183334    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183336    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183338    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183341    4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:05.197436 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183343    4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183346    4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183348    4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183350    4925 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183353    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183356    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183358    4925 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183361    4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183364    4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183366    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183368    4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183372    4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183374    4925 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183377    4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183379    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183381    4925 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183384    4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183386    4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183389    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183391    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:05.197930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183393    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183396    4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183400    4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183404    4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183406    4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183409    4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183411    4925 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183414    4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183416    4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183419    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183421    4925 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183424    4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183426    4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183429    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183431    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183433    4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183436    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183438    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183440    4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183443    4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:05.198423 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183445    4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183448    4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183462    4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183467    4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183471    4925 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183474    4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183477    4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183479    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183482    4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183484    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183487    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183489    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183492    4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183494    4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183497    4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183499    4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183501    4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183504    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183506    4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:05.198982 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183509    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183511    4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183513    4925 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183516    4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183518    4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183521    4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183523    4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183526    4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183528    4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183531    4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183533    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183538    4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183541    4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183543    4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183546    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183548    4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183551    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183553    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183556    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:05.199467 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183559    4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183561    4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.183583    4925 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183623    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183628    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183631    4925 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183633    4925 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183636    4925 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183638    4925 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183641    4925 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183644    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183646    4925 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183649    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183651    4925 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183654    4925 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183656    4925 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:06:05.199945 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183659    4925 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183661    4925 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183664    4925 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183666    4925 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183669    4925 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183671    4925 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183674    4925 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183676    4925 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183679    4925 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183698    4925 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183703    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183705    4925 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183708    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183710    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183713    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183716    4925 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183719    4925 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183721    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183724    4925 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:06:05.200329 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183727    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183729    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183732    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183734    4925 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183737    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183795    4925 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183813    4925 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183818    4925 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183822    4925 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183908    4925 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183914    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183920    4925 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183924    4925 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183929    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183934    4925 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183939    4925 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183943    4925 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183948    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183952    4925 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:06:05.200913 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183957    4925 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183961    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183972    4925 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183976    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183980    4925 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183985    4925 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183989    4925 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183993    4925 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.183998    4925 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184002    4925 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184007    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184012    4925 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184016    4925 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184020    4925 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184030    4925 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184034    4925 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184039    4925 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184043    4925 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184047    4925 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184052    4925 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:06:05.201367 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184057    4925 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184061    4925 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184066    4925 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184071    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184075    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184080    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184094    4925 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184101    4925 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184107    4925 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184112    4925 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184118    4925 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184122    4925 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184126    4925 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184130    4925 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184135    4925 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184139    4925 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184144    4925 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184148    4925 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184153    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184162    4925 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:06:05.201871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184167    4925 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184171    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184175    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184179    4925 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184184    4925 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184188    4925 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184192    4925 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184196    4925 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184203    4925 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184210    4925 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184214    4925 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184224    4925 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184229    4925 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184233    4925 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:05.184237    4925 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.184362    4925 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.186009    4925 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.186074    4925 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:06:05.202521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.186283    4925 server.go:1014] "Starting client certificate rotation"
Oct 30 19:06:05.203244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.186422    4925 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:06:05.203244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.186477    4925 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:00:24 +0000 UTC" deadline="2028-05-23 23:47:31.602830571 +0000 UTC"
Oct 30 19:06:05.203244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.186508    4925 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13708h41m26.416324945s"
Oct 30 19:06:05.203244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.186818    4925 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:06:05.203244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.187903    4925 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:06:05.203244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.189074    4925 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:06:05.203244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.191194    4925 log.go:25] "Validated CRI v1 image API"
Oct 30 19:06:05.203244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.191980    4925 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:06:05.203244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.193862    4925 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7c290a57-105a-4130-a477-f6225d60ac2f:/dev/nvme0n1p4 e99e1900-b542-4f09-aa3c-50b6724e5909:/dev/nvme0n1p3]
Oct 30 19:06:05.203542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.193880    4925 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0aa5593d5694eb972e93f7cc5e3448a8f2cfa0ab94a834102d7ad3574edb5a3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0aa5593d5694eb972e93f7cc5e3448a8f2cfa0ab94a834102d7ad3574edb5a3e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ab5ca8613b565b6e3fe91d5b4d5a1589fd58846713bda11f6c235b9307e34aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ab5ca8613b565b6e3fe91d5b4d5a1589fd58846713bda11f6c235b9307e34aa/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f4e84a81880dae7b48902ade6f1ec5a4d2708bc84b8755086f6cd3ea282ac49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f4e84a81880dae7b48902ade6f1ec5a4d2708bc84b8755086f6cd3ea282ac49/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/212f7152c441ccfa9301249e058e618d195e0747939018f1b8920fb518c32a63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/212f7152c441ccfa9301249e058e618d195e0747939018f1b8920fb518c32a63/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4722938f7e9c1ab3faeb57149c44afe1dac61833d8c5ecd2c8e3b400db486889/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4722938f7e9c1ab3faeb57149c44afe1dac61833d8c5ecd2c8e3b400db486889/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ceae37138053d042352d72c74d79c13e0a4ad093fce13ba773093779b24ca17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ceae37138053d042352d72c74d79c13e0a4ad093fce13ba773093779b24ca17/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53d5a4c69bd68265e94eadf485e8ca53248bf9c67a94fda25844f2e50f0fce13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53d5a4c69bd68265e94eadf485e8ca53248bf9c67a94fda25844f2e50f0fce13/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5520ce5889c47c068fb865cade61ffb9163be214d467761137a7e7d2365fafbe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5520ce5889c47c068fb865cade61ffb9163be214d467761137a7e7d2365fafbe/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58fc931b6298e1b64482a8c903d3d18ea79e704ece34764fe5767a47d0912302/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58fc931b6298e1b64482a8c903d3d18ea79e704ece34764fe5767a47d0912302/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/658dab09f57ff6bef530c98dc853d56069401b335461734b2440ba313e6ac6c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/658dab09f57ff6bef530c98dc853d56069401b335461734b2440ba313e6ac6c5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bff760d87968173196ea3e9a1fe4715736ae9884952a5494102135d1d1a8a5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bff760d87968173196ea3e9a1fe4715736ae9884952a5494102135d1d1a8a5c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95ad42f668a84230cc02f05dbf573a03b0e75be0300c6a1f7804f4277bc435d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95ad42f668a84230cc02f05dbf573a03b0e75be0300c6a1f7804f4277bc435d8/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/966f43d2e6e3083fd23b390f7f2671a59b0fcd6bef9029c8977239423567835e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/966f43d2e6e3083fd23b390f7f2671a59b0fcd6bef9029c8977239423567835e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9adaaeb07f102a51567ae75bb7e4ef9e99112c54befe35f0802e0a538024bdb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9adaaeb07f102a51567ae75bb7e4ef9e99112c54befe35f0802e0a538024bdb9/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5b3209e4da4d75788d2c6b68e9ee81bb41741aaad04068e91157cf5869a9c0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5b3209e4da4d75788d2c6b68e9ee81bb41741aaad04068e91157cf5869a9c0e/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a84e44f8ce51c5f5b9e483c2dfd6de795a7fee5a68e6880958423432bc5cb292/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a84e44f8ce51c5f5b9e483c2dfd6de795a7fee5a68e6880958423432bc5cb292/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1396fa35e406f4017536ce0642bb361155e0fff2b7ab427af13beb8a5edafb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1396fa35e406f4017536ce0642bb361155e0fff2b7ab427af13beb8a5edafb5/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbd8e2af697fc4b14f06e3bdf10ba7c5a19c4da2db8fa9a73aefa1831b1f9af6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbd8e2af697fc4b14f06e3bdf10ba7c5a19c4da2db8fa9a73aefa1831b1f9af6/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd83e00a593d27b08d553990bbab5434c286d2f7fe74a4b21ed5eec63d9dd25d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd83e00a593d27b08d553990bbab5434c286d2f7fe74a4b21ed5eec63d9dd25d/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df1e6b61bd4fd7b4b0d8cd476636fef4dd175e3ea6524a7ae5016c684c59ef8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df1e6b61bd4fd7b4b0d8cd476636fef4dd175e3ea6524a7ae5016c684c59ef8a/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed155d4bb9b9d4727d417f7d28c2c978e960cf563542b52a13b9b0ae958555c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed155d4bb9b9d4727d417f7d28c2c978e960cf563542b52a13b9b0ae958555c3/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8c8a827085254904a7915c0e10b86573f5d1c7f52481ac02f5af35046310c49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8c8a827085254904a7915c0e10b86573f5d1c7f52481ac02f5af35046310c49/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd46ba12cd184a7c2e2e90b6afb919b5584f69d7415d56b58adec7462304bb11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd46ba12cd184a7c2e2e90b6afb919b5584f69d7415d56b58adec7462304bb11/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0413837b-14e3-4465-8a24-1bca4f394e06/volumes/kubernetes.io~projected/kube-api-access-j5thr:{mountpoint:/var/lib/kubelet/pods/0413837b-14e3-4465-8a24-1bca4f394e06/volumes/kubernetes.io~projected/kube-api-access-j5thr major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/151fec2f-192b-4af4-9d4e-10a790100cb9/volumes/kubernetes.io~projected/kube-api-access-rf6tx:{mountpoint:/var/lib/kubelet/pods/151fec2f-192b-4af4-9d4e-10a790100cb9/volumes/kubernetes.io~projected/kube-api-access-rf6tx major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/151fec2f-192b-4af4-9d4e-10a790100cb9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/151fec2f-192b-4af4-9d4e-10a790100cb9/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1aaa292a-397b-4c16-a897-f4df8a4af32f/volumes/kubernetes.io~projected/kube-api-access-pf2b4:{mountpoint:/var/lib/kubelet/pods/1aaa292a-397b-4c16-a897-f4df8a4af32f/volumes/kubernetes.io~projected/kube-api-access-pf2b4 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0/volumes/kubernetes.io~projected/kube-api-access-9cwlx:{mountpoint:/var/lib/kubelet/pods/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0/volumes/kubernetes.io~projected/kube-api-access-9cwlx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec7ed63-0fc7-4c17-a301-a4035c43feef/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2ec7ed63-0fc7-4c17-a301-a4035c43feef/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec7ed63-0fc7-4c17-a301-a4035c43feef/volumes/kubernetes.io~projected/kube-api-access-hp2s8:{mountpoint:/var/lib/kubelet/pods/2ec7ed63-0fc7-4c17-a301-a4035c43feef/volumes/kubernetes.io~projected/kube-api-access-hp2s8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ec7ed63-0fc7-4c17-a301-a4035c43feef/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2ec7ed63-0fc7-4c17-a301-a4035c43feef/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/336005f6-92be-46c1-aa77-3fe7ab10970e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/336005f6-92be-46c1-aa77-3fe7ab10970e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/336005f6-92be-46c1-aa77-3fe7ab10970e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/336005f6-92be-46c1-aa77-3fe7ab10970e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/336005f6-92be-46c1-aa77-3fe7ab10970e/volumes/kubernetes.io~projected/kube-api-access-wb7pf:{mountpoint:/var/lib/kubelet/pods/336005f6-92be-46c1-aa77-3fe7ab10970e/volumes/kubernetes.io~projected/kube-api-access-wb7pf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c9100be-82b2-466c-833c-3ed05703fba2/volumes/kubernetes.io~projected/kube-api-access-wlpzx:{mountpoint:/var/lib/kubelet/pods/3c9100be-82b2-466c-833c-3ed05703fba2/volumes/kubernetes.io~projected/kube-api-access-wlpzx major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cbe6234-bffe-4a9f-a543-8c509b08357e/volumes/kubernetes.io~projected/kube-api-access-nrhsk:{mountpoint:/var/lib/kubelet/pods/3cbe6234-bffe-4a9f-a543-8c509b08357e/volumes/kubernetes.io~projected/kube-api-access-nrhsk major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e61d03d-c4b1-4ac3-8802-27d32191c403/volumes/kubernetes.io~projected/kube-api-access-szps7:{mountpoint:/var/lib/kubelet/pods/3e61d03d-c4b1-4ac3-8802-27d32191c403/volumes/kubernetes.io~projected/kube-api-access-szps7 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40c4bb90-1b90-4e9c-9822-5daafa73cdfa/volumes/kubernetes.io~projected/kube-api-access-55pfd:{mountpoint:/var/lib/kubelet/pods/40c4bb90-1b90-4e9c-9822-5daafa73cdfa/volumes/kubernetes.io~projected/kube-api-access-55pfd major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40c4bb90-1b90-4e9c-9822-5daafa73cdfa/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/40c4bb90-1b90-4e9c-9822-5daafa73cdfa/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45274ad9-273b-467a-8bf1-27a876267294/volumes/kubernetes.io~projected/kube-api-access-vtmbk:{mountpoint:/var/lib/kubelet/pods/45274ad9-273b-467a-8bf1-27a876267294/volumes/kubernetes.io~projected/kube-api-access-vtmbk major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5250b4fe-4b08-4a7a-bfc7-d7343f202585/volumes/kubernetes.io~projected/kube-api-access-v6js6:{mountpoint:/var/lib/kubelet/pods/5250b4fe-4b08-4a7a-bfc7-d7343f202585/volumes/kubernetes.io~projected/kube-api-access-v6js6 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/664daa2d-471e-4380-bfdc-73df3a83ab3e/volumes/kubernetes.io~projected/kube-api-access-wgl6d:{mountpoint:/var/lib/kubelet/pods/664daa2d-471e-4380-bfdc-73df3a83ab3e/volumes/kubernetes.io~projected/kube-api-access-wgl6d major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/664daa2d-471e-4380-bfdc-73df3a83ab3e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/664daa2d-471e-4380-bfdc-73df3a83ab3e/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6996469c-158c-46af-9e06-a89a9f7137c0/volumes/kubernetes.io~projected/kube-api-access-55hst:{mountpoint:/var/lib/kubelet/pods/6996469c-158c-46af-9e06-a89a9f7137c0/volumes/kubernetes.io~projected/kube-api-access-55hst major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a4b6b51-385b-4e07-9eae-5d57cd734d39/volumes/kubernetes.io~projected/kube-api-access-f8fjb:{mountpoint:/var/lib/kubelet/pods/6a4b6b51-385b-4e07-9eae-5d57cd734d39/volumes/kubernetes.io~projected/kube-api-access-f8fjb major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/797c4f9e-bf72-41af-a1df-f39270f68e1a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/797c4f9e-bf72-41af-a1df-f39270f68e1a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~projected/kube-api-access-rk967:{mountpoint:/var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~projected/kube-api-access-rk967 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c6e8d41-096c-4121-b9c9-a147a7ee010d/volumes/kubernetes.io~projected/kube-api-access-m46s4:{mountpoint:/var/lib/kubelet/pods/8c6e8d41-096c-4121-b9c9-a147a7ee010d/volumes/kubernetes.io~projected/kube-api-access-m46s4 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c6e8d41-096c-4121-b9c9-a147a7ee010d/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/8c6e8d41-096c-4121-b9c9-a147a7ee010d/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c6e8d41-096c-4121-b9c9-a147a7ee010d/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/8c6e8d41-096c-4121-b9c9-a147a7ee010d/volumes/kubernetes.io~secret/stats-auth major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f/volumes/kubernetes.io~projected/kube-api-access-kmqs8:{mountpoint:/var/lib/kubelet/pods/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f/volumes/kubernetes.io~projected/kube-api-access-kmqs8 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a69fe59-2039-43ff-a132-eeed44422412/volumes/kubernetes.io~projected/kube-api-access-xthsn:{mountpoint:/var/lib/kubelet/pods/9a69fe59-2039-43ff-a132-eeed44422412/volumes/kubernetes.io~projected/kube-api-access-xthsn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a09e9550-86c1-474b-9774-50dd9189f3fa/volumes/kubernetes.io~projected/kube-api-access-xk78m:{mountpoint:/var/lib/kubelet/pods/a09e9550-86c1-474b-9774-50dd9189f3fa/volumes/kubernetes.io~projected/kube-api-access-xk78m major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a09e9550-86c1-474b-9774-50dd9189f3fa/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a09e9550-86c1-474b-9774-50dd9189f3fa/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a50accc7-ce62-4f71-9a4a-aa128cfe10c4/volumes/kubernetes.io~projected/kube-api-access-6bdzq:{mountpoint:/var/lib/kubelet/pods/a50accc7-ce62-4f71-9a4a-aa128cfe10c4/volumes/kubernetes.io~projected/kube-api-access-6bdzq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be3e41d8-e9c9-49bb-af2b-3ccf8acab832/volumes/kubernetes.io~projected/kube-api-access-vjmkv:{mountpoint:/var/lib/kubelet/pods/be3e41d8-e9c9-49bb-af2b-3ccf8acab832/volumes/kubernetes.io~projected/kube-api-access-vjmkv major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~projected/kube-api-access-mnrvr:{mountpoint:/var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~projected/kube-api-access-mnrvr major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a95351-6d86-4411-903e-3514b69f6bb7/volumes/kubernetes.io~projected/kube-api-access-bb4ct:{mountpoint:/var/lib/kubelet/pods/c0a95351-6d86-4411-903e-3514b69f6bb7/volumes/kubernetes.io~projected/kube-api-access-bb4ct major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c91b7f8f-3427-4d04-a5bd-31792aed5075/volumes/kubernetes.io~projected/kube-api-access-gn4h5:{mountpoint:/var/lib/kubelet/pods/c91b7f8f-3427-4d04-a5bd-31792aed5075/volumes/kubernetes.io~projected/kube-api-access-gn4h5 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d11698df-8a69-42d9-96a9-ce0d5a37b003/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d11698df-8a69-42d9-96a9-ce0d5a37b003/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04/volumes/kubernetes.io~projected/kube-api-access-hxcpn:{mountpoint:/var/lib/kubelet/pods/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04/volumes/kubernetes.io~projected/kube-api-access-hxcpn major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6/volumes/kubernetes.io~projected/kube-api-access-t5d7d:{mountpoint:/var/lib/kubelet/pods/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6/volumes/kubernetes.io~projected/kube-api-access-t5d7d major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffc0076c-0b02-421c-851a-616ecf70e075/volumes/kubernetes.io~projected/kube-api-access-b5htb:{mountpoint:/var/lib/kubelet/pods/ffc0076c-0b02-421c-851a-616ecf70e075/volumes/kubernetes.io~projected/kube-api-access-b5htb major:0 minor:310 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c265f9230c091bda207d18f9212746601f3cc90fe6f2609f6a6f53e64d52e1d9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2299558f27883837873b335d963057780b0ca1a54b23c00b80ba0a14b096ef4f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/017cea3d25ec7f124afa7b0cd1b2b765524f218e585e73a2ab853d7233b04666/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b652534285a5c6b8284a414d00f33a2c787a33f828e1e847147f16ce3851b334/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6f40ecb835908474b7d4e4f4b186efa8e4ee975c9e269200b099cf3d842aed1e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/eac6b1549ff70c34bfa151a1066aa3e6f033568d6c8f45fc5030ba7c57865bb0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0f648280154ec6ecd5923f5a0e9e95072527a15d1ed15cf5d3b230740511d9eb/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a47c124985db492bddd031a7ccb8461e1ff4cc284369fd974c3956500a15dfb8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/23af5116b5e67b0c48df60dc602e3288ac7ea6085acc26e302fd903192461501/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/983dbdb1ca50b21774609993ae9223090fac7657aeefd89d19999b90920fed43/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/9d2d22e7fab3c4040c0b1af144a6b3695a4e8424d750327008dc785e990f6212/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/92cfab6446b23c33e075c8d99e96151835800c9d7dbfdd9e6d01f315dcf5a640/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0ecbf726302eb71c27d86eff1e93525b6c95287939022640a0da7024fd733be2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/896aa6abd38286cd746ad0439a09a53ef6436dbc0f43c2de8c26139b2a9d24bc/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/50a4d1f8d1e9490b8afa36789c814a8e4c8fa63ea56102d741bfc0734740125f/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b7c2dca21dd683bc81758118b539c4617a5872fb343189b9d2e901c5845d1c4d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8c8085473e94164947ab0870d2778b8db9e2ad41b64205e9f6a49216c6802bc7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/12e4814c7c7e4961ab1c22092d4d467d290bc53dab2e09cfe7ede9b6741c1e95/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c44011f9d5e63968c90499dd0dc0c415f4e3339a9e49403e0f7fcf42c850b6aa/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/df6fd3ab42df2b1b331988e4d8f9a69ce749e048e3ae96f6b314ea0d8fb2f95a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/98ae2ebe71b5b7f9b2a4a105c986a7b3f129d7445636bcb5d8394dfab5c7f97b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/ac49cba71a1dbe388246a06a61e2598318ca84e18338bc713298052a87a75cc0/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/c1af16f3d6271c75483dc71e361f2c7fa5e8dfa4e892fec7a9947b00a255a3f4/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/83290278215c7012b1807be03de833c1e84b6445833f5bd8a23f1cfce606b1a8/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/791bd476137e5d3d9d2e3c2585ade350c52c8846e0b1b32cf3b8e37d2ffb14e4/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9e69de797cf63eeffc81eee4928a8752dd77eeff6912c39a4bf7581a9026807d/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/30915d65d470d9146c0f870323bcc0046c18b8d05b45eed58c14e5d590a1e180/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/60512d2383b0c460863dc89a624ca32acc65d1483884710a521368005067733c/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/103dc7b79904bfb512bfc82f8006a2bfa3aba50ff3ee9d734cd5e22b94f46b27/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/4db3e10d83aa1b14c69e867c2ed3a477b28cc4a8bae55c172ce3318206183e9b/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/7235401f40326665f26b2bf0a11641049744388b9b15a4c81a2a0141ef2de7de/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/69f6dd9566bb934610ea9faf9d5701afd1a56ed01b053631ee4170b4a738b083/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/437b274834b97d3d0f02e26297f1f7299f08f55e691df8bf853abff8d023775c/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/1b607fa8ea325605f539960e175039cdd24ba66136093ab86ee8b3ebf767164e/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/7b2519ded574ce9de45d24e917a79a32d5bdf16324702045d0eb066c50527379/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/ad229cb885527a5e0de213760cc783cc7b190a4fe2717ed3a4b1cd6d7a069d78/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/e4f00d4059b83e8e40ae9d87c46f9df1a929e0a49a077371e40fd27729e27798/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/832bdbd6cd7273ccc8988f26ba58a651dac6cd2e1e8738528372450b93674021/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/31e7bfbd03e2b45f7e54d99159a9872a114ab932ccd1b51275f9850496d7abea/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/4537cf281260a9f223b6ec714edb11eb79243875808d8205f312395661e05067/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/e5a61a9065a3fdd52b6c35dc7b6236cf48a4774c40a5ae4cfa715f6e20c45018/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/eb6f0eb4faa281ded034a88819d8c4ef7c670791dad24a2d5cba9653607a3c7b/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/e10a7fae9fab2f67ef37c46f146e95ac9b2d3348a74e9c810cd5aecbac3c28a8/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/f272b7c860708963524758032b377fb032a9a44fbce72ea980c1dcb18d290d3d/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/783f3d9f9331c56a2311cf71953287ec5862bb0b39cda28e5bff77b592a8f36b/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/bbce8dd919c4c4487d7480db5937ced1eb0b275a12839b04dd131d7408098ffd/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/e45ff90a51b345f089117a4a3780ed15e9eedb37145dafdb8b6efbeafd5b43d2/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5b9a7f5ff652a4d64cdcb00786bab5766da1731d8f544fbb287b8b3c40896f14/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/080d51d94b2a99b1535ac9190833b470ebaf06a30dcc41db589444a212eca0a6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/079205a44bd375a31f17a4f7caea3569cad3e68690c7e88c20b8792fe1cea780/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/d6557087016fc0bce191092de7a5ed6634d804cdfe29030c1ea39446ba5b3601/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/acf3213ebbd375c41bbcd1856a0ea8f32060f3b73597c4b0f82fdc3f4eab942f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/114c3bd0b2e78b29d3d1551957933f7135d42aa3b9fe91a5606ce5895f1020c1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/cae23803b8a5ec6b3938feaf83c8d8a4786a8c3c859e61f1fe8c6be4f65dab7c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/dbbeb00ce279c6131232e7f00b3effe566db382276201120b4bf51223de642ff/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/316c96691344d0ab98eaf7e3107cd4ae6d6bc995d8f900b4decd5ddfb7817b18/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:06:05.204258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.203921    4925 manager.go:217] Machine: {Timestamp:2026-10-30 19:06:05.202806371 +0000 UTC m=+0.105819399 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109628 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e7d76f9dae2eaf75579ff7893a50d SystemUUID:ec2e7d76-f9da-e2ea-f755-79ff7893a50d BootID:d7498e4e-d21b-45d5-9f11-e046e22f5945 Filesystems:[{Device:/var/lib/kubelet/pods/c0a95351-6d86-4411-903e-3514b69f6bb7/volumes/kubernetes.io~projected/kube-api-access-bb4ct DeviceMajor:0 DeviceMinor:227 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a84e44f8ce51c5f5b9e483c2dfd6de795a7fee5a68e6880958423432bc5cb292/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5b3209e4da4d75788d2c6b68e9ee81bb41741aaad04068e91157cf5869a9c0e/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53d5a4c69bd68265e94eadf485e8ca53248bf9c67a94fda25844f2e50f0fce13/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec7ed63-0fc7-4c17-a301-a4035c43feef/volumes/kubernetes.io~projected/kube-api-access-hp2s8 DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ab5ca8613b565b6e3fe91d5b4d5a1589fd58846713bda11f6c235b9307e34aa/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8c6e8d41-096c-4121-b9c9-a147a7ee010d/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:206 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0/volumes/kubernetes.io~projected/kube-api-access-9cwlx DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~projected/kube-api-access-mnrvr DeviceMajor:0 DeviceMinor:213 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5250b4fe-4b08-4a7a-bfc7-d7343f202585/volumes/kubernetes.io~projected/kube-api-access-v6js6 DeviceMajor:0 DeviceMinor:219 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6996469c-158c-46af-9e06-a89a9f7137c0/volumes/kubernetes.io~projected/kube-api-access-55hst DeviceMajor:0 DeviceMinor:221 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbd8e2af697fc4b14f06e3bdf10ba7c5a19c4da2db8fa9a73aefa1831b1f9af6/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/797c4f9e-bf72-41af-a1df-f39270f68e1a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec7ed63-0fc7-4c17-a301-a4035c43feef/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bff760d87968173196ea3e9a1fe4715736ae9884952a5494102135d1d1a8a5c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c91b7f8f-3427-4d04-a5bd-31792aed5075/volumes/kubernetes.io~projected/kube-api-access-gn4h5 DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8c6e8d41-096c-4121-b9c9-a147a7ee010d/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/0413837b-14e3-4465-8a24-1bca4f394e06/volumes/kubernetes.io~projected/kube-api-access-j5thr DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/658dab09f57ff6bef530c98dc853d56069401b335461734b2440ba313e6ac6c5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a69fe59-2039-43ff-a132-eeed44422412/volumes/kubernetes.io~projected/kube-api-access-xthsn DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8c8a827085254904a7915c0e10b86573f5d1c7f52481ac02f5af35046310c49/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40c4bb90-1b90-4e9c-9822-5daafa73cdfa/volumes/kubernetes.io~projected/kube-api-access-55pfd DeviceMajor:0 DeviceMinor:215 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9adaaeb07f102a51567ae75bb7e4ef9e99112c54befe35f0802e0a538024bdb9/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ffc0076c-0b02-421c-851a-616ecf70e075/volumes/kubernetes.io~projected/kube-api-access-b5htb DeviceMajor:0 DeviceMinor:310 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/966f43d2e6e3083fd23b390f7f2671a59b0fcd6bef9029c8977239423567835e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~projected/kube-api-access-rk967 DeviceMajor:0 DeviceMinor:226 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd83e00a593d27b08d553990bbab5434c286d2f7fe74a4b21ed5eec63d9dd25d/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/336005f6-92be-46c1-aa77-3fe7ab10970e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d11698df-8a69-42d9-96a9-ce0d5a37b003/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40c4bb90-1b90-4e9c-9822-5daafa73cdfa/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/45274ad9-273b-467a-8bf1-27a876267294/volumes/kubernetes.io~projected/kube-api-access-vtmbk DeviceMajor:0 DeviceMinor:225 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04/volumes/kubernetes.io~projected/kube-api-access-hxcpn DeviceMajor:0 DeviceMinor:328 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/336005f6-92be-46c1-aa77-3fe7ab10970e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1aaa292a-397b-4c16-a897-f4df8a4af32f/volumes/kubernetes.io~projected/kube-api-access-pf2b4 DeviceMajor:0 DeviceMinor:217 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/664daa2d-471e-4380-bfdc-73df3a83ab3e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd46ba12cd184a7c2e2e90b6afb919b5584f69d7415d56b58adec7462304bb11/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/664daa2d-471e-4380-bfdc-73df3a83ab3e/volumes/kubernetes.io~projected/kube-api-access-wgl6d DeviceMajor:0 DeviceMinor:223 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1396fa35e406f4017536ce0642bb361155e0fff2b7ab427af13beb8a5edafb5/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0aa5593d5694eb972e93f7cc5e3448a8f2cfa0ab94a834102d7ad3574edb5a3e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2ec7ed63-0fc7-4c17-a301-a4035c43feef/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8c6e8d41-096c-4121-b9c9-a147a7ee010d/volumes/kubernetes.io~projected/kube-api-access-m46s4 DeviceMajor:0 DeviceMinor:216 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df1e6b61bd4fd7b4b0d8cd476636fef4dd175e3ea6524a7ae5016c684c59ef8a/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a4b6b51-385b-4e07-9eae-5d57cd734d39/volumes/kubernetes.io~projected/kube-api-access-f8fjb DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/336005f6-92be-46c1-aa77-3fe7ab10970e/volumes/kubernetes.io~projected/kube-api-access-wb7pf DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5520ce5889c47c068fb865cade61ffb9163be214d467761137a7e7d2365fafbe/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/212f7152c441ccfa9301249e058e618d195e0747939018f1b8920fb518c32a63/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a09e9550-86c1-474b-9774-50dd9189f3fa/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/151fec2f-192b-4af4-9d4e-10a790100cb9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95ad42f668a84230cc02f05dbf573a03b0e75be0300c6a1f7804f4277bc435d8/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a50accc7-ce62-4f71-9a4a-aa128cfe10c4/volumes/kubernetes.io~projected/kube-api-access-6bdzq DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f/volumes/kubernetes.io~projected/kube-api-access-kmqs8 DeviceMajor:0 DeviceMinor:218 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3cbe6234-bffe-4a9f-a543-8c509b08357e/volumes/kubernetes.io~projected/kube-api-access-nrhsk DeviceMajor:0 DeviceMinor:293 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58fc931b6298e1b64482a8c903d3d18ea79e704ece34764fe5767a47d0912302/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ceae37138053d042352d72c74d79c13e0a4ad093fce13ba773093779b24ca17/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a09e9550-86c1-474b-9774-50dd9189f3fa/volumes/kubernetes.io~projected/kube-api-access-xk78m DeviceMajor:0 DeviceMinor:222 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3e61d03d-c4b1-4ac3-8802-27d32191c403/volumes/kubernetes.io~projected/kube-api-access-szps7 DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed155d4bb9b9d4727d417f7d28c2c978e960cf563542b52a13b9b0ae958555c3/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f4e84a81880dae7b48902ade6f1ec5a4d2708bc84b8755086f6cd3ea282ac49/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6/volumes/kubernetes.io~projected/kube-api-access-t5d7d DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3c9100be-82b2-466c-833c-3ed05703fba2/volumes/kubernetes.io~projected/kube-api-access-wlpzx DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4722938f7e9c1ab3faeb57149c44afe1dac61833d8c5ecd2c8e3b400db486889/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/151fec2f-192b-4af4-9d4e-10a790100cb9/volumes/kubernetes.io~projected/kube-api-access-rf6tx DeviceMajor:0 DeviceMinor:224 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/be3e41d8-e9c9-49bb-af2b-3ccf8acab832/volumes/kubernetes.io~projected/kube-api-access-vjmkv DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1f4e84a81880dae MacAddress:b2:58:7f:3e:8b:5e Speed:10000 Mtu:8901} {Name:4722938f7e9c1ab MacAddress:2a:e3:f2:e7:c6:ec Speed:10000 Mtu:8901} {Name:53d5a4c69bd6826 MacAddress:f2:f1:4d:96:fd:f8 Speed:10000 Mtu:8901} {Name:58fc931b6298e1b MacAddress:92:5b:34:a8:f3:74 Speed:10000 Mtu:8901} {Name:95ad42f668a8423 MacAddress:f2:9c:1e:7d:ff:30 Speed:10000 Mtu:8901} {Name:9adaaeb07f102a5 MacAddress:7a:37:98:b7:1d:56 Speed:10000 Mtu:8901} {Name:a5b3209e4da4d75 MacAddress:da:e9:39:a3:7b:96 Speed:10000 Mtu:8901} {Name:a84e44f8ce51c5f MacAddress:3e:7d:66:ef:62:f6 Speed:10000 Mtu:8901} {Name:b1396fa35e406f4 MacAddress:5e:fa:22:8e:90:19 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e9:f8:1b:3b Speed:0 Mtu:9001} {Name:br-int MacAddress:4e:ae:49:e5:ce:c7 Speed:0 Mtu:8901} {Name:cbd8e2af697fc4b MacAddress:de:ad:8c:27:ca:e3 Speed:10000 Mtu:8901} {Name:cd83e00a593d27b MacAddress:6e:45:5c:56:46:84 Speed:10000 Mtu:8901} {Name:ed155d4bb9b9d47 MacAddress:fe:b6:6c:f2:77:53 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e9:f8:1b:3b Speed:0 Mtu:9001} {Name:fd46ba12cd184a7 MacAddress:e2:9c:33:c3:53:ca Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:da:fe:f7:df:7d:d1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:02:2e:f3:c9:99:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:06:05.204434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204262    4925 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:06:05.204434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204307    4925 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:06:05.204583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204556    4925 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:06:05.204798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204584    4925 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-164.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:06:05.204798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204732    4925 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:06:05.204798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204741    4925 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:06:05.204798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204752    4925 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:06:05.204798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204772    4925 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:06:05.204798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204790    4925 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:06:05.204967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.204938    4925 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:06:05.205041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.205032    4925 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:06:05.205077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.205069    4925 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:06:05.205107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.205085    4925 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:06:05.205107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.205095    4925 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:06:05.205107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.205103    4925 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:06:05.205200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.205111    4925 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:06:05.205759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.205745    4925 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:06:05.205759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.205757    4925 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:06:05.206223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.206200    4925 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:06:05.208329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208317    4925 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:06:05.208329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208330    4925 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:06:05.208431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208417    4925 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:06:05.208608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208597    4925 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:06:05.208642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208626    4925 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:06:05.208801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208792    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:06:05.208852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208808    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:06:05.208852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208814    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:06:05.208852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208820    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:06:05.208852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208826    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:06:05.208852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208831    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:06:05.208852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208837    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:06:05.208852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208842    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:06:05.208852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208848    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:06:05.208852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208854    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:06:05.209182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208867    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:06:05.209182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208876    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:06:05.209182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208900    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:06:05.209182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.208905    4925 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:06:05.212247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.212233    4925 server.go:1263] "Started kubelet"
Oct 30 19:06:05.212349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.212320    4925 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:06:05.212859 ip-10-0-130-164 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:06:05.214297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.214240    4925 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:06:05.214390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.214322    4925 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:06:05.214438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.214413    4925 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:06:05.214649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.214632    4925 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:06:05.218887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.218576    4925 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:06:05.218887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.218807    4925 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:06:05.218887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.218834    4925 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:06:05.218887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.218845    4925 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:06:05.219201 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:05.218876    4925 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-164.us-east-2.compute.internal\" not found"
Oct 30 19:06:05.219201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.218917    4925 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:06:05.219201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.218947    4925 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:00:26 +0000 UTC" deadline="2028-08-05 07:00:05.73221082 +0000 UTC"
Oct 30 19:06:05.219201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.218983    4925 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15467h54m0.513231083s"
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.219646    4925 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.219662    4925 factory.go:55] Registering systemd factory
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.219671    4925 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.220090    4925 factory.go:153] Registering CRI-O factory
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.220104    4925 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.220140    4925 factory.go:103] Registering Raw factory
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.220156    4925 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.220451    4925 manager.go:319] Starting recovery of all containers
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221184    4925 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221535    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45274ad9-273b-467a-8bf1-27a876267294" volumeName="kubernetes.io/projected/45274ad9-273b-467a-8bf1-27a876267294-kube-api-access-vtmbk" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221567    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151fec2f-192b-4af4-9d4e-10a790100cb9" volumeName="kubernetes.io/configmap/151fec2f-192b-4af4-9d4e-10a790100cb9-config" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221581    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b64c148-ae60-4adc-b3fa-f3b4aeb229d0" volumeName="kubernetes.io/projected/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kube-api-access-9cwlx" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221594    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c6e8d41-096c-4121-b9c9-a147a7ee010d" volumeName="kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-stats-auth" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221606    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d11698df-8a69-42d9-96a9-ce0d5a37b003" volumeName="kubernetes.io/secret/d11698df-8a69-42d9-96a9-ce0d5a37b003-agent-certs" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221619    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336005f6-92be-46c1-aa77-3fe7ab10970e" volumeName="kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-tmp" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221631    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e61d03d-c4b1-4ac3-8802-27d32191c403" volumeName="kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221642    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9d8478-9a05-4d2c-9210-a08764728647" volumeName="kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221656    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973eb9f4-c418-4ca6-9cb1-bd5b4980b83f" volumeName="kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221668    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a69fe59-2039-43ff-a132-eeed44422412" volumeName="kubernetes.io/configmap/9a69fe59-2039-43ff-a132-eeed44422412-serviceca" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221680    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9d8478-9a05-4d2c-9210-a08764728647" volumeName="kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221717    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c6e8d41-096c-4121-b9c9-a147a7ee010d" volumeName="kubernetes.io/projected/8c6e8d41-096c-4121-b9c9-a147a7ee010d-kube-api-access-m46s4" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221730    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a69fe59-2039-43ff-a132-eeed44422412" volumeName="kubernetes.io/projected/9a69fe59-2039-43ff-a132-eeed44422412-kube-api-access-xthsn" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221742    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a50accc7-ce62-4f71-9a4a-aa128cfe10c4" volumeName="kubernetes.io/projected/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-kube-api-access-6bdzq" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221759    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be3e41d8-e9c9-49bb-af2b-3ccf8acab832" volumeName="kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221773    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be3e41d8-e9c9-49bb-af2b-3ccf8acab832" volumeName="kubernetes.io/projected/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-kube-api-access-vjmkv" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221786    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9d8478-9a05-4d2c-9210-a08764728647" volumeName="kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221798    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336005f6-92be-46c1-aa77-3fe7ab10970e" volumeName="kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-tuned" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221810    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336005f6-92be-46c1-aa77-3fe7ab10970e" volumeName="kubernetes.io/projected/336005f6-92be-46c1-aa77-3fe7ab10970e-kube-api-access-wb7pf" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221820    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="664daa2d-471e-4380-bfdc-73df3a83ab3e" volumeName="kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-config" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221831    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9d8478-9a05-4d2c-9210-a08764728647" volumeName="kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221842    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e61d03d-c4b1-4ac3-8802-27d32191c403" volumeName="kubernetes.io/projected/3e61d03d-c4b1-4ac3-8802-27d32191c403-kube-api-access-szps7" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221853    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9d8478-9a05-4d2c-9210-a08764728647" volumeName="kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221864    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c6e8d41-096c-4121-b9c9-a147a7ee010d" volumeName="kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-default-certificate" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221875    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973eb9f4-c418-4ca6-9cb1-bd5b4980b83f" volumeName="kubernetes.io/projected/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-kube-api-access-kmqs8" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221899    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c08977e9-bc18-4071-8748-2c6d7bc94f80" volumeName="kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221909    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c08977e9-bc18-4071-8748-2c6d7bc94f80" volumeName="kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221919    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151fec2f-192b-4af4-9d4e-10a790100cb9" volumeName="kubernetes.io/projected/151fec2f-192b-4af4-9d4e-10a790100cb9-kube-api-access-rf6tx" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221935    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5250b4fe-4b08-4a7a-bfc7-d7343f202585" volumeName="kubernetes.io/empty-dir/5250b4fe-4b08-4a7a-bfc7-d7343f202585-tmp-dir" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221946    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6996469c-158c-46af-9e06-a89a9f7137c0" volumeName="kubernetes.io/configmap/6996469c-158c-46af-9e06-a89a9f7137c0-telemetry-config" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221958    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973eb9f4-c418-4ca6-9cb1-bd5b4980b83f" volumeName="kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221986    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9e20809-cbf6-491c-b6e9-371655cbd3c6" volumeName="kubernetes.io/configmap/c9e20809-cbf6-491c-b6e9-371655cbd3c6-nginx-conf" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.221998    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d11698df-8a69-42d9-96a9-ce0d5a37b003" volumeName="kubernetes.io/configmap/d11698df-8a69-42d9-96a9-ce0d5a37b003-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222010    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a09e9550-86c1-474b-9774-50dd9189f3fa" volumeName="kubernetes.io/projected/a09e9550-86c1-474b-9774-50dd9189f3fa-kube-api-access-xk78m" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222023    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc72d076-0c85-4722-a5fe-3c20dcd0b0e6" volumeName="kubernetes.io/configmap/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222035    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5250b4fe-4b08-4a7a-bfc7-d7343f202585" volumeName="kubernetes.io/projected/5250b4fe-4b08-4a7a-bfc7-d7343f202585-kube-api-access-v6js6" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222047    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec7ed63-0fc7-4c17-a301-a4035c43feef" volumeName="kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-env-overrides" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222059    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec7ed63-0fc7-4c17-a301-a4035c43feef" volumeName="kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-config" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222070    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec7ed63-0fc7-4c17-a301-a4035c43feef" volumeName="kubernetes.io/projected/2ec7ed63-0fc7-4c17-a301-a4035c43feef-kube-api-access-hp2s8" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222081    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9100be-82b2-466c-833c-3ed05703fba2" volumeName="kubernetes.io/projected/3c9100be-82b2-466c-833c-3ed05703fba2-kube-api-access-wlpzx" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222094    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="797c4f9e-bf72-41af-a1df-f39270f68e1a" volumeName="kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222107    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973eb9f4-c418-4ca6-9cb1-bd5b4980b83f" volumeName="kubernetes.io/secret/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-serving-cert" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222118    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c08977e9-bc18-4071-8748-2c6d7bc94f80" volumeName="kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222130    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c4bb90-1b90-4e9c-9822-5daafa73cdfa" volumeName="kubernetes.io/projected/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-kube-api-access-55pfd" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222141    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec7ed63-0fc7-4c17-a301-a4035c43feef" volumeName="kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222153    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c4bb90-1b90-4e9c-9822-5daafa73cdfa" volumeName="kubernetes.io/configmap/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-config" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222172    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="664daa2d-471e-4380-bfdc-73df3a83ab3e" volumeName="kubernetes.io/secret/664daa2d-471e-4380-bfdc-73df3a83ab3e-serving-cert" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222185    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c08977e9-bc18-4071-8748-2c6d7bc94f80" volumeName="kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222198    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c91b7f8f-3427-4d04-a5bd-31792aed5075" volumeName="kubernetes.io/projected/c91b7f8f-3427-4d04-a5bd-31792aed5075-kube-api-access-gn4h5" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222212    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffc0076c-0b02-421c-851a-616ecf70e075" volumeName="kubernetes.io/projected/ffc0076c-0b02-421c-851a-616ecf70e075-kube-api-access-b5htb" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222224    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="664daa2d-471e-4380-bfdc-73df3a83ab3e" volumeName="kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-trusted-ca" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222235    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4b6b51-385b-4e07-9eae-5d57cd734d39" volumeName="kubernetes.io/projected/6a4b6b51-385b-4e07-9eae-5d57cd734d39-kube-api-access-f8fjb" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222248    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be3e41d8-e9c9-49bb-af2b-3ccf8acab832" volumeName="kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222261    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c08977e9-bc18-4071-8748-2c6d7bc94f80" volumeName="kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222287    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c08977e9-bc18-4071-8748-2c6d7bc94f80" volumeName="kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222298    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e61d03d-c4b1-4ac3-8802-27d32191c403" volumeName="kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222320    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9d8478-9a05-4d2c-9210-a08764728647" volumeName="kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222332    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0413837b-14e3-4465-8a24-1bca4f394e06" volumeName="kubernetes.io/projected/0413837b-14e3-4465-8a24-1bca4f394e06-kube-api-access-j5thr" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222345    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973eb9f4-c418-4ca6-9cb1-bd5b4980b83f" volumeName="kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-snapshots" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222357    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c08977e9-bc18-4071-8748-2c6d7bc94f80" volumeName="kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222369    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a95351-6d86-4411-903e-3514b69f6bb7" volumeName="kubernetes.io/projected/c0a95351-6d86-4411-903e-3514b69f6bb7-kube-api-access-bb4ct" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222381    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04" volumeName="kubernetes.io/empty-dir/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-data-volume" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222393    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc72d076-0c85-4722-a5fe-3c20dcd0b0e6" volumeName="kubernetes.io/projected/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-kube-api-access-t5d7d" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222406    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a09e9550-86c1-474b-9774-50dd9189f3fa" volumeName="kubernetes.io/secret/a09e9550-86c1-474b-9774-50dd9189f3fa-cert" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222421    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40c4bb90-1b90-4e9c-9822-5daafa73cdfa" volumeName="kubernetes.io/secret/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-serving-cert" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222434    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cbe6234-bffe-4a9f-a543-8c509b08357e" volumeName="kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222446    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e61d03d-c4b1-4ac3-8802-27d32191c403" volumeName="kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222458    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="664daa2d-471e-4380-bfdc-73df3a83ab3e" volumeName="kubernetes.io/projected/664daa2d-471e-4380-bfdc-73df3a83ab3e-kube-api-access-wgl6d" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222469    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151fec2f-192b-4af4-9d4e-10a790100cb9" volumeName="kubernetes.io/secret/151fec2f-192b-4af4-9d4e-10a790100cb9-serving-cert" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222481    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ec7ed63-0fc7-4c17-a301-a4035c43feef" volumeName="kubernetes.io/secret/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222493    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9d8478-9a05-4d2c-9210-a08764728647" volumeName="kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222507    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04" volumeName="kubernetes.io/projected/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-api-access-hxcpn" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222520    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1aaa292a-397b-4c16-a897-f4df8a4af32f" volumeName="kubernetes.io/projected/1aaa292a-397b-4c16-a897-f4df8a4af32f-kube-api-access-pf2b4" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222533    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5250b4fe-4b08-4a7a-bfc7-d7343f202585" volumeName="kubernetes.io/configmap/5250b4fe-4b08-4a7a-bfc7-d7343f202585-config-volume" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222545    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6996469c-158c-46af-9e06-a89a9f7137c0" volumeName="kubernetes.io/projected/6996469c-158c-46af-9e06-a89a9f7137c0-kube-api-access-55hst" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222558    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973eb9f4-c418-4ca6-9cb1-bd5b4980b83f" volumeName="kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-tmp" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222574    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a50accc7-ce62-4f71-9a4a-aa128cfe10c4" volumeName="kubernetes.io/empty-dir/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-tmp-dir" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222584    4925 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a95351-6d86-4411-903e-3514b69f6bb7" volumeName="kubernetes.io/configmap/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-config" seLinuxMountContext=""
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222596    4925 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:06:05.222593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222604    4925 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:06:05.226468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.222996    4925 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:06:05.226468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.223612    4925 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:06:05.226468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.225656    4925 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:06:05.236638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.236547    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/1.log"
Oct 30 19:06:05.240185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.240166    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/1.log"
Oct 30 19:06:05.252073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.252031    4925 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:06:05.254391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.254369    4925 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:06:05.254476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.254393    4925 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:06:05.254476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.254413    4925 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:06:05.254594 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:05.254469    4925 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:06:05.259540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.259516    4925 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:06:05.284231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.284201    4925 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="1768dda1663bcbebde47fdf5e26cca988c55bac174b4af9c4db90256ed4db797" exitCode=0
Oct 30 19:06:05.284231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.284227    4925 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="be350b506d74dbb956d05949e613759050195985c812feeb4fb0a767b608128a" exitCode=0
Oct 30 19:06:05.284231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.284236    4925 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="01a7238a5e0c089bb848e5c8b172f7817eec47b98adf7ba7d10643e5cf521b21" exitCode=0
Oct 30 19:06:05.284448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.284244    4925 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="0d9f26846bf8d876f4cb9ec83cc19495066ed24ae9f32ee8436dd52749fb3378" exitCode=0
Oct 30 19:06:05.284448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.284255    4925 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="1bff2c2ae6d66f7fb1ea6524984adcaa52f51b85f52d023e457a33840c6f0317" exitCode=0
Oct 30 19:06:05.284448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.284263    4925 generic.go:356] "Generic (PLEG): container finished" podID="3e61d03d-c4b1-4ac3-8802-27d32191c403" containerID="658a05489150d7963d2f5af1274e0d6ed15db43629d80bdb4d57e2836c8b5c3b" exitCode=0
Oct 30 19:06:05.299661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.299587    4925 generic.go:356] "Generic (PLEG): container finished" podID="a08b19af43d4124fc88d192e2bced995" containerID="57b558318dd96fd8c462c97a798aea9ded01191d083d980a347bce8ef8bcc118" exitCode=0
Oct 30 19:06:05.304357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.304338    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/1.log"
Oct 30 19:06:05.304443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.304365    4925 generic.go:356] "Generic (PLEG): container finished" podID="664daa2d-471e-4380-bfdc-73df3a83ab3e" containerID="1e3532dddac09b5586753aa2a69e1243fc6678c9ea895299412cdbb3a073465b" exitCode=255
Oct 30 19:06:05.314579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.314560    4925 manager.go:324] Recovery completed
Oct 30 19:06:05.335479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.335457    4925 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:06:05.335479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.335471    4925 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:06:05.335616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.335488    4925 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:06:05.335616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.335612    4925 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:06:05.336068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.335622    4925 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:06:05.336104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.336070    4925 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:06:05.336104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.336077    4925 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:06:05.336104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.336083    4925 policy_none.go:50] "Start"
Oct 30 19:06:05.336104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.336099    4925 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:06:05.336230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.336110    4925 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:06:05.336258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.336242    4925 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:06:05.336258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.336249    4925 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:06:05.336258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.336254    4925 policy_none.go:44] "Start"
Oct 30 19:06:05.340565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.340526    4925 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:06:05.343207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.343191    4925 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:06:05.343302 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:05.343214    4925 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:06:05.343302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.343223    4925 server.go:87] "Starting device plugin registration server"
Oct 30 19:06:05.343523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.343508    4925 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:06:05.343593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.343523    4925 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:06:05.343649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.343639    4925 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:06:05.343757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.343732    4925 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:06:05.343824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.343764    4925 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:06:05.343824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.343771    4925 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:06:05.343920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.343830    4925 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:06:05.343754788Z","UUID":"9a656893-e4c8-48da-b970-b526bb8751ba","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:06:05.345177 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:05.345151    4925 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:06:05.345473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.345455    4925 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:06:05.345473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.345477    4925 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:06:05.354584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.354546    4925 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"]
Oct 30 19:06:05.355347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.355326    4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.355438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.355397    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"a08b19af43d4124fc88d192e2bced995","Type":"ContainerStarted","Data":"3f47b662d4261a67d3bda8c864bcfcc383c98f5749b16cae714bcffbbbd63ac5"}
Oct 30 19:06:05.355496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.355438    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"a08b19af43d4124fc88d192e2bced995","Type":"ContainerDied","Data":"57b558318dd96fd8c462c97a798aea9ded01191d083d980a347bce8ef8bcc118"}
Oct 30 19:06:05.355496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.355459    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"a08b19af43d4124fc88d192e2bced995","Type":"ContainerStarted","Data":"966f43d2e6e3083fd23b390f7f2671a59b0fcd6bef9029c8977239423567835e"}
Oct 30 19:06:05.355496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.355490    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"64f9ae4e88f69ebe611e7b19f38b4cc6","Type":"ContainerStarted","Data":"7c506c34fc646f3580e8bcb8fea0902f8a68f7d6bd0770e9f1a1129875d95f18"}
Oct 30 19:06:05.355668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.355508    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal" event={"ID":"64f9ae4e88f69ebe611e7b19f38b4cc6","Type":"ContainerStarted","Data":"212f7152c441ccfa9301249e058e618d195e0747939018f1b8920fb518c32a63"}
Oct 30 19:06:05.355668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.355541    4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.378852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.378830    4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:06:05.378942 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:05.378872    4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.379781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.379768    4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:06:05.379838 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:05.379801    4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.427083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.427060    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/64f9ae4e88f69ebe611e7b19f38b4cc6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"64f9ae4e88f69ebe611e7b19f38b4cc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.427178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.427089    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.427178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.427114    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.444178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.444159    4925 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:06:05.445311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.445295    4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:06:05.445375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.445323    4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:06:05.445375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.445335    4925 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-164.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:06:05.445445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.445381    4925 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.477366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.477346    4925 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.477436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.477406    4925 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.527654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.527627    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.527788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.527706    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.527788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.527727    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.527788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.527742    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a08b19af43d4124fc88d192e2bced995-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"a08b19af43d4124fc88d192e2bced995\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.527788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.527772    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/64f9ae4e88f69ebe611e7b19f38b4cc6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"64f9ae4e88f69ebe611e7b19f38b4cc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:05.527940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:05.527820    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/64f9ae4e88f69ebe611e7b19f38b4cc6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal\" (UID: \"64f9ae4e88f69ebe611e7b19f38b4cc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:06.205267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.205235    4925 apiserver.go:52] "Watching apiserver"
Oct 30 19:06:06.214521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.214496    4925 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:06:06.215136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.215114    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kj2xn","openshift-network-console/networking-console-plugin-86c478bb88-wqlmz","openshift-ingress/router-default-ddbc459d5-jmrh9","openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7","openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj","openshift-ovn-kubernetes/ovnkube-node-cgjj4","openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz","kube-system/kas-connection-checker-685895c7c9-jkwnj","kube-system/konnectivity-agent-24ddg","openshift-ingress-canary/ingress-canary-nb5x6","openshift-multus/multus-additional-cni-plugins-h9ktv","openshift-multus/network-metrics-daemon-882bp","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp","openshift-network-operator/iptables-alerter-29spv","openshift-image-registry/image-registry-bd99d66bc-r6hq2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t","openshift-insights/insights-runtime-extractor-x4lg8","openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal","openshift-multus/multus-mz8rn","kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-b8m22","openshift-dns/dns-default-4dcpr","openshift-image-registry/node-ca-cnsll","openshift-insights/insights-operator-7c4986d449-h6hw8","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j","openshift-dns/node-resolver-g76xc","openshift-network-diagnostics/network-check-target-9s7mb","kube-system/kas-connection-checker-685895c7c9-2xnmc","openshift-image-registry/image-registry-775fb78d4b-vms7p","kube-system/kas-connection-checker-685895c7c9-98bt4","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh","openshift-console-operator/console-operator-7cd56b8dbc-fskzc"]
Oct 30 19:06:06.216964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.216947    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:06.217674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.217657    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:06.218851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.218834    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.219358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.219340    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:06:06.219358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.219348    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.219358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.219348    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.219943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.219930    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.220504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.220489    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:06.222504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.222485    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:06.222789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.222769    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.222872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.222788    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.222872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.222839    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.222872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.222839    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.222872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.222860    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.223079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.222878    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:06:06.223079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.222879    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.223079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.222936    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.223079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.223067    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.223277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.223094    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.223277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.223116    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.223277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.223245    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.223514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.223489    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.223612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.223522    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:06:06.223612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.223559    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.224848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.224833    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:06:06.225530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.225509    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:06:06.226043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.226023    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9b8n4\""
Oct 30 19:06:06.226118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.226092    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rdn2g\""
Oct 30 19:06:06.226861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.226828    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:06:06.227059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.227042    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:06:06.227528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.227509    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:06:06.227757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.227742    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7gl8m\""
Oct 30 19:06:06.228396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.228349    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hfhz5\""
Oct 30 19:06:06.228756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.228740    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wfls2\""
Oct 30 19:06:06.228930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.228916    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:06:06.229101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.229086    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:06:06.229429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.229411    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:06:06.229780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.229765    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:06:06.230128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.230110    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:06:06.231114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231090    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-serving-cert\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.231205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231131    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:06.231205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231160    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.231205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231202    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fjb\" (UniqueName: \"kubernetes.io/projected/6a4b6b51-385b-4e07-9eae-5d57cd734d39-kube-api-access-f8fjb\") pod \"kas-connection-checker-685895c7c9-2xnmc\" (UID: \"6a4b6b51-385b-4e07-9eae-5d57cd734d39\") " pod="kube-system/kas-connection-checker-685895c7c9-2xnmc"
Oct 30 19:06:06.231361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231253    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgl6d\" (UniqueName: \"kubernetes.io/projected/664daa2d-471e-4380-bfdc-73df3a83ab3e-kube-api-access-wgl6d\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:06.231361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231321    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-config\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:06:06.231494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231468    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:06:06.231556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231490    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/151fec2f-192b-4af4-9d4e-10a790100cb9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:06:06.231606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231591    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:06:06.231649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231620    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:06:06.231724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231710    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hsp67\""
Oct 30 19:06:06.231774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231745    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/151fec2f-192b-4af4-9d4e-10a790100cb9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:06:06.231822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231770    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-serving-cert\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.231822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231799    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:06:06.231822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231803    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6tx\" (UniqueName: \"kubernetes.io/projected/151fec2f-192b-4af4-9d4e-10a790100cb9-kube-api-access-rf6tx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:06:06.231966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231779    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ml89v\""
Oct 30 19:06:06.231966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231862    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4h5\" (UniqueName: \"kubernetes.io/projected/c91b7f8f-3427-4d04-a5bd-31792aed5075-kube-api-access-gn4h5\") pod \"kas-connection-checker-685895c7c9-jkwnj\" (UID: \"c91b7f8f-3427-4d04-a5bd-31792aed5075\") " pod="kube-system/kas-connection-checker-685895c7c9-jkwnj"
Oct 30 19:06:06.231966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231876    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-stgll\""
Oct 30 19:06:06.231966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231893    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-snapshots\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.231966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231922    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.231966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231925    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-service-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.231966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231951    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xd9sf\""
Oct 30 19:06:06.231966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231957    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.231966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231964    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-frsmn\""
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.231989    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-stats-auth\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232007    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232024    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232054    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232070    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232177    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-snapshots\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232225    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232258    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232022    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6996469c-158c-46af-9e06-a89a9f7137c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232282    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-service-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232298    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ktwjf\""
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232305    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/664daa2d-471e-4380-bfdc-73df3a83ab3e-serving-cert\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:06.232357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232340    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-trusted-ca\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232369    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232375    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/c0a95351-6d86-4411-903e-3514b69f6bb7-kube-api-access-bb4ct\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232406    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pfd\" (UniqueName: \"kubernetes.io/projected/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-kube-api-access-55pfd\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232421    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232435    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2b4\" (UniqueName: \"kubernetes.io/projected/1aaa292a-397b-4c16-a897-f4df8a4af32f-kube-api-access-pf2b4\") pod \"network-check-source-6f78d59f99-6d9pz\" (UID: \"1aaa292a-397b-4c16-a897-f4df8a4af32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232449    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232463    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232493    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46s4\" (UniqueName: \"kubernetes.io/projected/8c6e8d41-096c-4121-b9c9-a147a7ee010d-kube-api-access-m46s4\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232524    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlpzx\" (UniqueName: \"kubernetes.io/projected/3c9100be-82b2-466c-833c-3ed05703fba2-kube-api-access-wlpzx\") pod \"kas-connection-checker-685895c7c9-98bt4\" (UID: \"3c9100be-82b2-466c-833c-3ed05703fba2\") " pod="kube-system/kas-connection-checker-685895c7c9-98bt4"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232534    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-m74bb\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232546    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232547    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-stats-auth\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232554    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmbk\" (UniqueName: \"kubernetes.io/projected/45274ad9-273b-467a-8bf1-27a876267294-kube-api-access-vtmbk\") pod \"volume-data-source-validator-58c9f5f46d-fndkp\" (UID: \"45274ad9-273b-467a-8bf1-27a876267294\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232584    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-tmp\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232604    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232614    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232648    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232664    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232612    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hst\" (UniqueName: \"kubernetes.io/projected/6996469c-158c-46af-9e06-a89a9f7137c0-kube-api-access-55hst\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232712    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232737    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-config\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232776    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232798    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232801    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232802    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151fec2f-192b-4af4-9d4e-10a790100cb9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232836    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmqs8\" (UniqueName: \"kubernetes.io/projected/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-kube-api-access-kmqs8\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232863    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-default-certificate\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232869    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/664daa2d-471e-4380-bfdc-73df3a83ab3e-serving-cert\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232890    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-serving-cert\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232920    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232345    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232935    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vt4hq\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232947    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:06:06.232964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232469    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233000    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233051    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233175    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233222    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tc7v2\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233234    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-config\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233351    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-default-certificate\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232869    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233450    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232307    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6996469c-158c-46af-9e06-a89a9f7137c0-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233551    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-serving-cert\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233593    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-tmp\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.232948    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233616    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/151fec2f-192b-4af4-9d4e-10a790100cb9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233741    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-trztf\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233741    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233814    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fljvz\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.233886    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.234303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.234145    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:06:06.234987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.234353    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:06:06.234987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.234383    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:06:06.235118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.235096    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xlxzb\""
Oct 30 19:06:06.235649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.235634    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.236493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.236473    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jks97\""
Oct 30 19:06:06.236570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.236496    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:06:06.236640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.236628    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.236848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.236832    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.236848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.236841    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:06:06.236960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.236869    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.236995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.236976    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gkzrx\""
Oct 30 19:06:06.237853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.237835    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.238627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.238609    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.239655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.239643    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4z6gp\""
Oct 30 19:06:06.239722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.239658    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ngr9b\""
Oct 30 19:06:06.241801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.241788    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-config\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:06:06.242538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242516    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgl6d\" (UniqueName: \"kubernetes.io/projected/664daa2d-471e-4380-bfdc-73df3a83ab3e-kube-api-access-wgl6d\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:06.242632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242594    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:06:06.242632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242624    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:06:06.242740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242656    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:06:06.242790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242769    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242898    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242906    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242999    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243032    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xxr6f\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242924    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6h85f\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242937    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242915    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242943    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243000    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243085    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fjb\" (UniqueName: \"kubernetes.io/projected/6a4b6b51-385b-4e07-9eae-5d57cd734d39-kube-api-access-f8fjb\") pod \"kas-connection-checker-685895c7c9-2xnmc\" (UID: \"6a4b6b51-385b-4e07-9eae-5d57cd734d39\") " pod="kube-system/kas-connection-checker-685895c7c9-2xnmc"
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.242912    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243065    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hst\" (UniqueName: \"kubernetes.io/projected/6996469c-158c-46af-9e06-a89a9f7137c0-kube-api-access-55hst\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243142    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w2h58\""
Oct 30 19:06:06.243376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243356    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:06:06.243992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243703    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6qlhx\""
Oct 30 19:06:06.243992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243742    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6tx\" (UniqueName: \"kubernetes.io/projected/151fec2f-192b-4af4-9d4e-10a790100cb9-kube-api-access-rf6tx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hft8j\" (UID: \"151fec2f-192b-4af4-9d4e-10a790100cb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j"
Oct 30 19:06:06.243992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243752    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/664daa2d-471e-4380-bfdc-73df3a83ab3e-trusted-ca\") pod \"console-operator-7cd56b8dbc-fskzc\" (UID: \"664daa2d-471e-4380-bfdc-73df3a83ab3e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:06.243992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243790    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.243992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243903    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fsc2n\""
Oct 30 19:06:06.243992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.243991    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:06:06.244178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.244077    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-89jvk\""
Oct 30 19:06:06.247149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.247128    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmbk\" (UniqueName: \"kubernetes.io/projected/45274ad9-273b-467a-8bf1-27a876267294-kube-api-access-vtmbk\") pod \"volume-data-source-validator-58c9f5f46d-fndkp\" (UID: \"45274ad9-273b-467a-8bf1-27a876267294\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fndkp"
Oct 30 19:06:06.247365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.247353    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46s4\" (UniqueName: \"kubernetes.io/projected/8c6e8d41-096c-4121-b9c9-a147a7ee010d-kube-api-access-m46s4\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.248191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.248171    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlpzx\" (UniqueName: \"kubernetes.io/projected/3c9100be-82b2-466c-833c-3ed05703fba2-kube-api-access-wlpzx\") pod \"kas-connection-checker-685895c7c9-98bt4\" (UID: \"3c9100be-82b2-466c-833c-3ed05703fba2\") " pod="kube-system/kas-connection-checker-685895c7c9-98bt4"
Oct 30 19:06:06.248291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.248267    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4h5\" (UniqueName: \"kubernetes.io/projected/c91b7f8f-3427-4d04-a5bd-31792aed5075-kube-api-access-gn4h5\") pod \"kas-connection-checker-685895c7c9-jkwnj\" (UID: \"c91b7f8f-3427-4d04-a5bd-31792aed5075\") " pod="kube-system/kas-connection-checker-685895c7c9-jkwnj"
Oct 30 19:06:06.250751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.250185    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmqs8\" (UniqueName: \"kubernetes.io/projected/973eb9f4-c418-4ca6-9cb1-bd5b4980b83f-kube-api-access-kmqs8\") pod \"insights-operator-7c4986d449-h6hw8\" (UID: \"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f\") " pod="openshift-insights/insights-operator-7c4986d449-h6hw8"
Oct 30 19:06:06.253470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.253450    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pfd\" (UniqueName: \"kubernetes.io/projected/40c4bb90-1b90-4e9c-9822-5daafa73cdfa-kube-api-access-55pfd\") pod \"service-ca-operator-544d749c8c-stxf7\" (UID: \"40c4bb90-1b90-4e9c-9822-5daafa73cdfa\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7"
Oct 30 19:06:06.253861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.253840    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2b4\" (UniqueName: \"kubernetes.io/projected/1aaa292a-397b-4c16-a897-f4df8a4af32f-kube-api-access-pf2b4\") pod \"network-check-source-6f78d59f99-6d9pz\" (UID: \"1aaa292a-397b-4c16-a897-f4df8a4af32f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6d9pz"
Oct 30 19:06:06.253908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.253858    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4ct\" (UniqueName: \"kubernetes.io/projected/c0a95351-6d86-4411-903e-3514b69f6bb7-kube-api-access-bb4ct\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:06.284537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.284514    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:06.310860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.310837    4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:06.311015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.311000    4925 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:06.321242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.321221    4925 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:06:06.324554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.324539    4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:06:06.324621 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.324579    4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:06.325433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.325418    4925 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:06:06.325492 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.325454    4925 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-164.us-east-2.compute.internal"
Oct 30 19:06:06.333136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333120    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d11698df-8a69-42d9-96a9-ce0d5a37b003-konnectivity-ca\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:06:06.333194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333145    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.333194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333164    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.333194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333180    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cnibin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.333363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333199    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5thr\" (UniqueName: \"kubernetes.io/projected/0413837b-14e3-4465-8a24-1bca4f394e06-kube-api-access-j5thr\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:06.333363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333327    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-systemd-units\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.333363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333356    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.333503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333374    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.333503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333399    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.333503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333417    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d11698df-8a69-42d9-96a9-ce0d5a37b003-konnectivity-ca\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:06:06.333503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333436    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9e20809-cbf6-491c-b6e9-371655cbd3c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:06.333503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333468    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-binary-copy\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.333503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333496    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.333849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333510    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.333849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333549    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:06:06.333849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333566    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.333849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333594    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:06:06.333849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333626    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9e20809-cbf6-491c-b6e9-371655cbd3c6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:06.333849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333680    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.333849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333782    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-binary-copy\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.333849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333805    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a69fe59-2039-43ff-a132-eeed44422412-serviceca\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:06:06.333849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333842    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/797c4f9e-bf72-41af-a1df-f39270f68e1a-original-pull-secret\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333895    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-kubernetes\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.333937    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334050    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk967\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334082    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-hosts-file\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334107    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-slash\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334121    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334123    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a69fe59-2039-43ff-a132-eeed44422412-serviceca\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334141    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a69fe59-2039-43ff-a132-eeed44422412-host\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334197    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-tuned\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334228    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:06.334250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334254    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334304    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334308    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-tuned\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334345    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-kubelet\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334362    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-systemd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334394    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334418    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-sys-fs\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334446    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-sys\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334471    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334520    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334547    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-conf-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334564    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-daemon-config\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334579    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmkv\" (UniqueName: \"kubernetes.io/projected/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-kube-api-access-vjmkv\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334603    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-system-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334626    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-dbus\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334649    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334664    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-multus\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334679    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-iptables-alerter-script\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:06:06.334731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334717    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-var-lib-kubelet\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334776    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334780    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334823    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/ffc0076c-0b02-421c-851a-616ecf70e075-kube-api-access-b5htb\") pod \"migrator-66df5d496b-lqgxj\" (UID: \"ffc0076c-0b02-421c-851a-616ecf70e075\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334856    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-env-overrides\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334873    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-daemon-config\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334937    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-iptables-alerter-script\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.334963    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-script-lib\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335035    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5d7d\" (UniqueName: \"kubernetes.io/projected/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-kube-api-access-t5d7d\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335072    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-systemd\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335093    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-env-overrides\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335165    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-script-lib\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335172    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-kubelet\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335201    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335233    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-tmp-dir\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335284    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-tmp-dir\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:06:06.335326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335281    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-conf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335348    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2s8\" (UniqueName: \"kubernetes.io/projected/2ec7ed63-0fc7-4c17-a301-a4035c43feef-kube-api-access-hp2s8\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335429    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335459    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335482    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-kubelet-config\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335499    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335515    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335539    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-var-lib-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.335582    4925 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335704    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09e9550-86c1-474b-9774-50dd9189f3fa-cert\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335739    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-run\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335770    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335704    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.335817    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:06.835781643 +0000 UTC m=+1.738794659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : secret "router-metrics-certs-default" not found
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335868    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-bin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335873    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a09e9550-86c1-474b-9774-50dd9189f3fa-cert\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335896    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-multus-certs\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335939    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7pf\" (UniqueName: \"kubernetes.io/projected/336005f6-92be-46c1-aa77-3fe7ab10970e-kube-api-access-wb7pf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.335945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335947    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e61d03d-c4b1-4ac3-8802-27d32191c403-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.335980    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336092    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-lib-modules\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.336150    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:06.836136226 +0000 UTC m=+1.739149247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336188    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-tmp\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336212    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5250b4fe-4b08-4a7a-bfc7-d7343f202585-config-volume\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336235    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336240    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336005f6-92be-46c1-aa77-3fe7ab10970e-tmp\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336263    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-etc-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336290    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrvr\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336319    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxcpn\" (UniqueName: \"kubernetes.io/projected/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-api-access-hxcpn\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336346    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-hostroot\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336371    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336421    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5250b4fe-4b08-4a7a-bfc7-d7343f202585-config-volume\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336500    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336572    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-socket-dir-parent\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336580    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336593    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.336657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336610    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szps7\" (UniqueName: \"kubernetes.io/projected/3e61d03d-c4b1-4ac3-8802-27d32191c403-kube-api-access-szps7\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336726    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovn-node-metrics-cert\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336815    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336818    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-registration-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336847    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-netns\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336881    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-host\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336909    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-system-cni-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336949    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-etc-kubernetes\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.336973    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-bin\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337006    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337023    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-log-socket\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337064    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-data-volume\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.337074    4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337089    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-socket-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337105    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysconfig\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.337111    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls podName:6996469c-158c-46af-9e06-a89a9f7137c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:06.837102645 +0000 UTC m=+1.740115661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rmqbj" (UID: "6996469c-158c-46af-9e06-a89a9f7137c0") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337117    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovn-node-metrics-cert\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337133    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337161    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-data-volume\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337173    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d11698df-8a69-42d9-96a9-ce0d5a37b003-agent-certs\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337201    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-config\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337251    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337361    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cni-binary-copy\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337389    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-k8s-cni-cncf-io\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337415    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337416    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ec7ed63-0fc7-4c17-a301-a4035c43feef-ovnkube-config\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337425    4925 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337464    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d11698df-8a69-42d9-96a9-ce0d5a37b003-agent-certs\") pod \"konnectivity-agent-24ddg\" (UID: \"d11698df-8a69-42d9-96a9-ce0d5a37b003\") " pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:06:06.337489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337485    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337516    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-host-slash\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337548    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-crio-socket\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337565    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cni-binary-copy\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337600    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdzq\" (UniqueName: \"kubernetes.io/projected/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-kube-api-access-6bdzq\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337628    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xthsn\" (UniqueName: \"kubernetes.io/projected/9a69fe59-2039-43ff-a132-eeed44422412-kube-api-access-xthsn\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337790    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337822    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337866    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337934    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5250b4fe-4b08-4a7a-bfc7-d7343f202585-tmp-dir\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.337998    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6js6\" (UniqueName: \"kubernetes.io/projected/5250b4fe-4b08-4a7a-bfc7-d7343f202585-kube-api-access-v6js6\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338016    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5250b4fe-4b08-4a7a-bfc7-d7343f202585-tmp-dir\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338026    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-node-log\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338023    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338128    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-netd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338157    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-device-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338228    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwlx\" (UniqueName: \"kubernetes.io/projected/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kube-api-access-9cwlx\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338263    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-modprobe-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338289    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-cnibin\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338312    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-ovn\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338376    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk78m\" (UniqueName: \"kubernetes.io/projected/a09e9550-86c1-474b-9774-50dd9189f3fa-kube-api-access-xk78m\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338397    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-netns\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.338635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.338414    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:06.339629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.339614    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0a95351-6d86-4411-903e-3514b69f6bb7-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xhlzh\" (UID: \"c0a95351-6d86-4411-903e-3514b69f6bb7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:06.345747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.345728    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.345747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.345737    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmkv\" (UniqueName: \"kubernetes.io/projected/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-kube-api-access-vjmkv\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.345874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.345858    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5thr\" (UniqueName: \"kubernetes.io/projected/0413837b-14e3-4465-8a24-1bca4f394e06-kube-api-access-j5thr\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:06.345974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.345956    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.345974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.345967    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk967\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.349460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.349430    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2s8\" (UniqueName: \"kubernetes.io/projected/2ec7ed63-0fc7-4c17-a301-a4035c43feef-kube-api-access-hp2s8\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.349616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.349442    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrhsk\" (UniqueName: \"kubernetes.io/projected/3cbe6234-bffe-4a9f-a543-8c509b08357e-kube-api-access-nrhsk\") pod \"network-check-target-9s7mb\" (UID: \"3cbe6234-bffe-4a9f-a543-8c509b08357e\") " pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:06:06.349811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.349797    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/ffc0076c-0b02-421c-851a-616ecf70e075-kube-api-access-b5htb\") pod \"migrator-66df5d496b-lqgxj\" (UID: \"ffc0076c-0b02-421c-851a-616ecf70e075\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lqgxj"
Oct 30 19:06:06.350834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.350815    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szps7\" (UniqueName: \"kubernetes.io/projected/3e61d03d-c4b1-4ac3-8802-27d32191c403-kube-api-access-szps7\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.354129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.354111    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7pf\" (UniqueName: \"kubernetes.io/projected/336005f6-92be-46c1-aa77-3fe7ab10970e-kube-api-access-wb7pf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.356546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.356526    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrvr\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.356669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.356644    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxcpn\" (UniqueName: \"kubernetes.io/projected/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-api-access-hxcpn\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.356867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.356853    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5d7d\" (UniqueName: \"kubernetes.io/projected/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-kube-api-access-t5d7d\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:06:06.358318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.358300    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdzq\" (UniqueName: \"kubernetes.io/projected/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-kube-api-access-6bdzq\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:06:06.364220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.364204    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6js6\" (UniqueName: \"kubernetes.io/projected/5250b4fe-4b08-4a7a-bfc7-d7343f202585-kube-api-access-v6js6\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.364337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.364323    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwlx\" (UniqueName: \"kubernetes.io/projected/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kube-api-access-9cwlx\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.365255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.365238    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xthsn\" (UniqueName: \"kubernetes.io/projected/9a69fe59-2039-43ff-a132-eeed44422412-kube-api-access-xthsn\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:06:06.369379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.369360    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk78m\" (UniqueName: \"kubernetes.io/projected/a09e9550-86c1-474b-9774-50dd9189f3fa-kube-api-access-xk78m\") pod \"ingress-canary-nb5x6\" (UID: \"a09e9550-86c1-474b-9774-50dd9189f3fa\") " pod="openshift-ingress-canary/ingress-canary-nb5x6"
Oct 30 19:06:06.439650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439624    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.439769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439655    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-kubelet\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.439769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439679    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-systemd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.439769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439742    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.439769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439757    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.439769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439767    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-sys-fs\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439756    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-kubelet\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439781    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-systemd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439788    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-sys\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439803    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439825    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439842    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-sys\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439850    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-sys-fs\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439863    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-conf-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439845    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-conf-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.439919    4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.439966    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls podName:5250b4fe-4b08-4a7a-bfc7-d7343f202585 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:06.939954623 +0000 UTC m=+1.842967642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls") pod "dns-default-4dcpr" (UID: "5250b4fe-4b08-4a7a-bfc7-d7343f202585") : secret "dns-default-metrics-tls" not found
Oct 30 19:06:06.439989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439978    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-system-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.439998    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-dbus\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440016    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-multus\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440039    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-var-lib-kubelet\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440044    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-system-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440056    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-systemd\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440076    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-multus\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440081    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-dbus\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440086    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-kubelet\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440108    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-systemd\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440115    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-conf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440100    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-var-lib-kubelet\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440142    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-kubelet\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440146    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440179    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-kubelet-config\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440207    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-conf\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.440214    4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.440227    4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd99d66bc-r6hq2: secret "image-registry-tls" not found
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440244    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-var-lib-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440240    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/797c4f9e-bf72-41af-a1df-f39270f68e1a-kubelet-config\") pod \"global-pull-secret-syncer-kj2xn\" (UID: \"797c4f9e-bf72-41af-a1df-f39270f68e1a\") " pod="kube-system/global-pull-secret-syncer-kj2xn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440213    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-var-lib-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.440289    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls podName:c08977e9-bc18-4071-8748-2c6d7bc94f80 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:06.940276125 +0000 UTC m=+1.843289156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls") pod "image-registry-bd99d66bc-r6hq2" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80") : secret "image-registry-tls" not found
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440315    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-run\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440334    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-bin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440350    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-multus-certs\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440371    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-lib-modules\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.440377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440388    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-etc-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440411    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-var-lib-cni-bin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440411    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-hostroot\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440434    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-multus-certs\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440446    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-etc-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440453    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440449    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-hostroot\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440476    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440482    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-run\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440499    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-socket-dir-parent\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440540    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-lib-modules\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440546    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-ovn-kubernetes\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440551    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-socket-dir-parent\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.440559    4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.440572    4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-775fb78d4b-vms7p: secret "image-registry-tls" not found
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440577    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-registration-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440601    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-netns\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.440613    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls podName:8b9d8478-9a05-4d2c-9210-a08764728647 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:06.940597821 +0000 UTC m=+1.843610850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls") pod "image-registry-775fb78d4b-vms7p" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647") : secret "image-registry-tls" not found
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440631    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-run-netns\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440639    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-host\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440658    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-system-cni-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440669    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-registration-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440676    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-host\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440675    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-etc-kubernetes\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440728    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-system-cni-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440738    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-bin\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440750    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-etc-kubernetes\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440774    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-log-socket\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440779    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-bin\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440799    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-socket-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440802    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-log-socket\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440816    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysconfig\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440847    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-socket-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440912    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-k8s-cni-cncf-io\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.440840    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-k8s-cni-cncf-io\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.441213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441211    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441263    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-host-slash\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441272    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysconfig\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441292    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-crio-socket\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441329    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441332    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc72d076-0c85-4722-a5fe-3c20dcd0b0e6-host-slash\") pod \"iptables-alerter-29spv\" (UID: \"dc72d076-0c85-4722-a5fe-3c20dcd0b0e6\") " pod="openshift-network-operator/iptables-alerter-29spv"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441354    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441396    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441389    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-node-log\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441422    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-crio-socket\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441465    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-openvswitch\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441510    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-node-log\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441522    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-sysctl-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441539    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-netd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441575    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-device-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441603    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-modprobe-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441628    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-cni-netd\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441646    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-device-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441677    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-cnibin\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441714    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-modprobe-d\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441721    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-ovn\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441746    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-run-ovn\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441757    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-netns\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441788    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441823    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441846    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cnibin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441882    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-systemd-units\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441916    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441945    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.441945    4925 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441975    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442007    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-kubernetes\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442038    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-hosts-file\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442047    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-host-run-netns\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442067    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-slash\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442098    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a69fe59-2039-43ff-a132-eeed44422412-host\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442129    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442179    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-cnibin\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.442252    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert podName:c9e20809-cbf6-491c-b6e9-371655cbd3c6 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:06.942233301 +0000 UTC m=+1.845246321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wqlmz" (UID: "c9e20809-cbf6-491c-b6e9-371655cbd3c6") : secret "networking-console-plugin-cert" not found
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.441781    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e61d03d-c4b1-4ac3-8802-27d32191c403-cnibin\") pod \"multus-additional-cni-plugins-h9ktv\" (UID: \"3e61d03d-c4b1-4ac3-8802-27d32191c403\") " pod="openshift-multus/multus-additional-cni-plugins-h9ktv"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442305    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/336005f6-92be-46c1-aa77-3fe7ab10970e-etc-kubernetes\") pod \"tuned-b8m22\" (UID: \"336005f6-92be-46c1-aa77-3fe7ab10970e\") " pod="openshift-cluster-node-tuning-operator/tuned-b8m22"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442324    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be3e41d8-e9c9-49bb-af2b-3ccf8acab832-multus-cni-dir\") pod \"multus-mz8rn\" (UID: \"be3e41d8-e9c9-49bb-af2b-3ccf8acab832\") " pod="openshift-multus/multus-mz8rn"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442374    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a50accc7-ce62-4f71-9a4a-aa128cfe10c4-hosts-file\") pod \"node-resolver-g76xc\" (UID: \"a50accc7-ce62-4f71-9a4a-aa128cfe10c4\") " pod="openshift-dns/node-resolver-g76xc"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.442390    4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442411    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b64c148-ae60-4adc-b3fa-f3b4aeb229d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5t42t\" (UID: \"2b64c148-ae60-4adc-b3fa-f3b4aeb229d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t42t"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.442446    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:06.942434606 +0000 UTC m=+1.845447637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.442457    4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442490    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-systemd-units\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442480    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ec7ed63-0fc7-4c17-a301-a4035c43feef-host-slash\") pod \"ovnkube-node-cgjj4\" (UID: \"2ec7ed63-0fc7-4c17-a301-a4035c43feef\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.442536    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a69fe59-2039-43ff-a132-eeed44422412-host\") pod \"node-ca-cnsll\" (UID: \"9a69fe59-2039-43ff-a132-eeed44422412\") " pod="openshift-image-registry/node-ca-cnsll"
Oct 30 19:06:06.442673 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.442544    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:06.942507378 +0000 UTC m=+1.845520409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : secret "metrics-daemon-secret" not found
Oct 30 19:06:06.444805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.443080    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.518349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.518296    4925 scope.go:122] "RemoveContainer" containerID="1e3532dddac09b5586753aa2a69e1243fc6678c9ea895299412cdbb3a073465b"
Oct 30 19:06:06.518432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.518349    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"
Oct 30 19:06:06.567667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.567631    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:06.675085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.675063    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g76xc_a50accc7-ce62-4f71-9a4a-aa128cfe10c4/dns-node-resolver/0.log"
Oct 30 19:06:06.846159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.846089    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.846159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.846128    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:06.846361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.846182    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:06.846361 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.846239    4925 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:06:06.846361 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.846290    4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:06.846361 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.846299    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.846285191 +0000 UTC m=+2.749298222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : secret "router-metrics-certs-default" not found
Oct 30 19:06:06.846361 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.846323    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.846309297 +0000 UTC m=+2.749322311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:06:06.846361 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.846345    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls podName:6996469c-158c-46af-9e06-a89a9f7137c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.846336304 +0000 UTC m=+2.749349319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rmqbj" (UID: "6996469c-158c-46af-9e06-a89a9f7137c0") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:06.852612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.852578    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh"]
Oct 30 19:06:06.907941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.907926    4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:06:06.947537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.947518    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:06.947643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.947588    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:06.947725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.947664    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:06.947725 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947679    4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:06.947725 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947712    4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd99d66bc-r6hq2: secret "image-registry-tls" not found
Oct 30 19:06:06.947725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.947719    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947742    4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947755    4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-775fb78d4b-vms7p: secret "image-registry-tls" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947763    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls podName:c08977e9-bc18-4071-8748-2c6d7bc94f80 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.947743441 +0000 UTC m=+2.850756478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls") pod "image-registry-bd99d66bc-r6hq2" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80") : secret "image-registry-tls" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947796    4925 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947804    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls podName:8b9d8478-9a05-4d2c-9210-a08764728647 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.947788587 +0000 UTC m=+2.850801648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls") pod "image-registry-775fb78d4b-vms7p" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647") : secret "image-registry-tls" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947824    4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947848    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert podName:c9e20809-cbf6-491c-b6e9-371655cbd3c6 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.947823578 +0000 UTC m=+2.850836593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wqlmz" (UID: "c9e20809-cbf6-491c-b6e9-371655cbd3c6") : secret "networking-console-plugin-cert" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947870    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.947856102 +0000 UTC m=+2.850869121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947873    4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.947908    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.947892146 +0000 UTC m=+2.850905195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : secret "metrics-daemon-secret" not found
Oct 30 19:06:06.947912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.947824    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:06.948231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:06.947941    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:06.948231 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.948033    4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:06:06.948231 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:06.948065    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls podName:5250b4fe-4b08-4a7a-bfc7-d7343f202585 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.94805565 +0000 UTC m=+2.851068692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls") pod "dns-default-4dcpr" (UID: "5250b4fe-4b08-4a7a-bfc7-d7343f202585") : secret "dns-default-metrics-tls" not found
Oct 30 19:06:07.075630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.075601    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:07.315654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.315621    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh" event={"ID":"c0a95351-6d86-4411-903e-3514b69f6bb7","Type":"ContainerStarted","Data":"9d49f4ee317d0e1c579ee94aa3396ee3a6f009cbeaf5a23cf2d2154da64886db"}
Oct 30 19:06:07.317181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.317163    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:06:07.317558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.317543    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/1.log"
Oct 30 19:06:07.317627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.317578    4925 generic.go:356] "Generic (PLEG): container finished" podID="664daa2d-471e-4380-bfdc-73df3a83ab3e" containerID="785b1dc18fe0bbc67e07f4d9d1da6815ba204f86fed8e0c4f931dbcb956f6d3d" exitCode=255
Oct 30 19:06:07.317677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.317651    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" event={"ID":"664daa2d-471e-4380-bfdc-73df3a83ab3e","Type":"ContainerDied","Data":"785b1dc18fe0bbc67e07f4d9d1da6815ba204f86fed8e0c4f931dbcb956f6d3d"}
Oct 30 19:06:07.317771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.317710    4925 scope.go:122] "RemoveContainer" containerID="1e3532dddac09b5586753aa2a69e1243fc6678c9ea895299412cdbb3a073465b"
Oct 30 19:06:07.318236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.318220    4925 scope.go:122] "RemoveContainer" containerID="785b1dc18fe0bbc67e07f4d9d1da6815ba204f86fed8e0c4f931dbcb956f6d3d"
Oct 30 19:06:07.318373 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.318356    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fskzc_openshift-console-operator(664daa2d-471e-4380-bfdc-73df3a83ab3e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" podUID="664daa2d-471e-4380-bfdc-73df3a83ab3e"
Oct 30 19:06:07.671301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.671272    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cnsll_9a69fe59-2039-43ff-a132-eeed44422412/node-ca/0.log"
Oct 30 19:06:07.681055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.681025    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:07.856338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.856308    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:07.856338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.856342    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:07.856530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.856382    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:07.856530 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.856469    4925 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:06:07.856530 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.856490    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.85647143 +0000 UTC m=+4.759484467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : configmap references non-existent config key: service-ca.crt
Oct 30 19:06:07.856530 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.856529    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.856512341 +0000 UTC m=+4.759525371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : secret "router-metrics-certs-default" not found
Oct 30 19:06:07.856733 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.856555    4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:07.856733 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.856656    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls podName:6996469c-158c-46af-9e06-a89a9f7137c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.856638949 +0000 UTC m=+4.759651971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rmqbj" (UID: "6996469c-158c-46af-9e06-a89a9f7137c0") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:07.888580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.888551    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:07.957774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.957680    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:07.957774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.957761    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:07.957979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.957826    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:07.957979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.957849    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:07.957979 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.957854    4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:07.957979 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.957877    4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd99d66bc-r6hq2: secret "image-registry-tls" not found
Oct 30 19:06:07.957979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.957893    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:07.957979 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.957928    4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:07.957979 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.957948    4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-775fb78d4b-vms7p: secret "image-registry-tls" not found
Oct 30 19:06:07.957979 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.957974    4925 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:06:07.957979 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.957935    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls podName:c08977e9-bc18-4071-8748-2c6d7bc94f80 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.957914563 +0000 UTC m=+4.860927595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls") pod "image-registry-bd99d66bc-r6hq2" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80") : secret "image-registry-tls" not found
Oct 30 19:06:07.958448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.958008    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:07.958448 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.958014    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs podName:0413837b-14e3-4465-8a24-1bca4f394e06 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.958000811 +0000 UTC m=+4.861013826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs") pod "network-metrics-daemon-882bp" (UID: "0413837b-14e3-4465-8a24-1bca4f394e06") : secret "metrics-daemon-secret" not found
Oct 30 19:06:07.958448 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.958037    4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:07.958448 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.958062    4925 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:06:07.958448 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.958098    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.958076796 +0000 UTC m=+4.861089813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:07.958448 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.958121    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls podName:5250b4fe-4b08-4a7a-bfc7-d7343f202585 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.958109172 +0000 UTC m=+4.861122196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls") pod "dns-default-4dcpr" (UID: "5250b4fe-4b08-4a7a-bfc7-d7343f202585") : secret "dns-default-metrics-tls" not found
Oct 30 19:06:07.958448 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:07.958201    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls podName:8b9d8478-9a05-4d2c-9210-a08764728647 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:09.958128879 +0000 UTC m=+4.861141894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls") pod "image-registry-775fb78d4b-vms7p" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647") : secret "image-registry-tls" not found
Oct 30 19:06:07.960877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:07.960849    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9e20809-cbf6-491c-b6e9-371655cbd3c6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wqlmz\" (UID: \"c9e20809-cbf6-491c-b6e9-371655cbd3c6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:08.021211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.021181    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"
Oct 30 19:06:08.021211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.021202    4925 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:08.072453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.072358    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nb5x6_a09e9550-86c1-474b-9774-50dd9189f3fa/serve-healthcheck-canary/0.log"
Oct 30 19:06:08.248628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.248553    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:06:08.272858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.272825    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:08.298282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.298250    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wqlmz"]
Oct 30 19:06:08.302991 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:08.302955    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e20809_cbf6_491c_b6e9_371655cbd3c6.slice/crio-564d2399d911bb2b7ffd71cae9655b08e3ea2bb4368921616203e83960069a8a WatchSource:0}: Error finding container 564d2399d911bb2b7ffd71cae9655b08e3ea2bb4368921616203e83960069a8a: Status 404 returned error can't find the container with id 564d2399d911bb2b7ffd71cae9655b08e3ea2bb4368921616203e83960069a8a
Oct 30 19:06:08.323414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.323382    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:06:08.323815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.323773    4925 scope.go:122] "RemoveContainer" containerID="785b1dc18fe0bbc67e07f4d9d1da6815ba204f86fed8e0c4f931dbcb956f6d3d"
Oct 30 19:06:08.324002 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:08.323976    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fskzc_openshift-console-operator(664daa2d-471e-4380-bfdc-73df3a83ab3e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" podUID="664daa2d-471e-4380-bfdc-73df3a83ab3e"
Oct 30 19:06:08.324936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.324915    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz" event={"ID":"c9e20809-cbf6-491c-b6e9-371655cbd3c6","Type":"ContainerStarted","Data":"564d2399d911bb2b7ffd71cae9655b08e3ea2bb4368921616203e83960069a8a"}
Oct 30 19:06:08.471539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.471507    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lqgxj_ffc0076c-0b02-421c-851a-616ecf70e075/migrator/0.log"
Oct 30 19:06:08.485922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.485895    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:08.873541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:08.873470    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-hft8j_151fec2f-192b-4af4-9d4e-10a790100cb9/kube-storage-version-migrator-operator/0.log"
Oct 30 19:06:09.101087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.101033    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:09.330526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.330493    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wqlmz" event={"ID":"c9e20809-cbf6-491c-b6e9-371655cbd3c6","Type":"ContainerStarted","Data":"3eccc242beb7509e176fa4457fa2882cee4e6f8ee7ada0a41ba20f5f20abe798"}
Oct 30 19:06:09.332357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.332334    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh" event={"ID":"c0a95351-6d86-4411-903e-3514b69f6bb7","Type":"ContainerStarted","Data":"858399d235475020196ea32c2f4a9c56d1415e993b34589857869366e31a24ab"}
Oct 30 19:06:09.332454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.332385    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xhlzh" event={"ID":"c0a95351-6d86-4411-903e-3514b69f6bb7","Type":"ContainerStarted","Data":"28af84432e67ccb06ca9bdd9fbf29778722c3ba1d93daa4da6bf70167f7cc661"}
Oct 30 19:06:09.670769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.670733    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:09.868548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.868467    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:09.876476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.876450    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:09.876595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.876485    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:09.876595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.876519    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:09.876721 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.876606    4925 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:09.876721 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.876607    4925 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:06:09.876721 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.876660    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls podName:6996469c-158c-46af-9e06-a89a9f7137c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:13.876648084 +0000 UTC m=+8.779661098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-rmqbj" (UID: "6996469c-158c-46af-9e06-a89a9f7137c0") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:06:09.876721 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.876673    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs podName:8c6e8d41-096c-4121-b9c9-a147a7ee010d nodeName:}" failed. No retries permitted until 2026-10-30 19:06:13.876666534 +0000 UTC m=+8.779679550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs") pod "router-default-ddbc459d5-jmrh9" (UID: "8c6e8d41-096c-4121-b9c9-a147a7ee010d") : secret "router-metrics-certs-default" not found
Oct 30 19:06:09.877050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.877032    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c6e8d41-096c-4121-b9c9-a147a7ee010d-service-ca-bundle\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:09.977295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.977270    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:09.977437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.977363    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:09.977437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.977410    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:09.977437 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.977420    4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:09.977437 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.977437    4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-775fb78d4b-vms7p: secret "image-registry-tls" not found
Oct 30 19:06:09.977622 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.977498    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls podName:8b9d8478-9a05-4d2c-9210-a08764728647 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:13.977476858 +0000 UTC m=+8.880489876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls") pod "image-registry-775fb78d4b-vms7p" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647") : secret "image-registry-tls" not found
Oct 30 19:06:09.977622 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.977511    4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:09.977622 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.977566    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:13.977549395 +0000 UTC m=+8.880562414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:09.977622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.977437    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:09.977884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.977631    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:09.977884 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.977767    4925 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:06:09.977884 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.977780    4925 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd99d66bc-r6hq2: secret "image-registry-tls" not found
Oct 30 19:06:09.977884 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:09.977816    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls podName:c08977e9-bc18-4071-8748-2c6d7bc94f80 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:13.977804619 +0000 UTC m=+8.880817634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls") pod "image-registry-bd99d66bc-r6hq2" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80") : secret "image-registry-tls" not found
Oct 30 19:06:09.979914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.979886    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5250b4fe-4b08-4a7a-bfc7-d7343f202585-metrics-tls\") pod \"dns-default-4dcpr\" (UID: \"5250b4fe-4b08-4a7a-bfc7-d7343f202585\") " pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:09.980003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:09.979960    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0413837b-14e3-4465-8a24-1bca4f394e06-metrics-certs\") pod \"network-metrics-daemon-882bp\" (UID: \"0413837b-14e3-4465-8a24-1bca4f394e06\") " pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:10.123123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:10.123037    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-882bp"
Oct 30 19:06:10.137290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:10.137255    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:10.271463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:10.271433    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:10.337362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:10.337330    4925 generic.go:356] "Generic (PLEG): container finished" podID="973eb9f4-c418-4ca6-9cb1-bd5b4980b83f" containerID="28732040500bcbecd4c6ba544d087b5f2b4351809934d0ad4b78ab0a87440952" exitCode=0
Oct 30 19:06:10.337806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:10.337414    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-h6hw8" event={"ID":"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f","Type":"ContainerDied","Data":"28732040500bcbecd4c6ba544d087b5f2b4351809934d0ad4b78ab0a87440952"}
Oct 30 19:06:10.337919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:10.337902    4925 scope.go:122] "RemoveContainer" containerID="28732040500bcbecd4c6ba544d087b5f2b4351809934d0ad4b78ab0a87440952"
Oct 30 19:06:10.394300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:10.394266    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-882bp"]
Oct 30 19:06:10.398857 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:10.398824    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0413837b_14e3_4465_8a24_1bca4f394e06.slice/crio-913b48ccc6e0454f111fb4e67d4e19d3312408ad43fe742add3acb3f6e1d6bfb WatchSource:0}: Error finding container 913b48ccc6e0454f111fb4e67d4e19d3312408ad43fe742add3acb3f6e1d6bfb: Status 404 returned error can't find the container with id 913b48ccc6e0454f111fb4e67d4e19d3312408ad43fe742add3acb3f6e1d6bfb
Oct 30 19:06:10.415990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:10.415963    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4dcpr"]
Oct 30 19:06:10.420089 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:10.420068    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5250b4fe_4b08_4a7a_bfc7_d7343f202585.slice/crio-9df211b9dc9ffb2dbfe9de0317c32f4685bfafdd2a24c783ffff34d1b4298128 WatchSource:0}: Error finding container 9df211b9dc9ffb2dbfe9de0317c32f4685bfafdd2a24c783ffff34d1b4298128: Status 404 returned error can't find the container with id 9df211b9dc9ffb2dbfe9de0317c32f4685bfafdd2a24c783ffff34d1b4298128
Oct 30 19:06:10.469675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:10.469638    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:11.079765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:11.079710    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:11.250295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:11.250237    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:11.343311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:11.343219    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-h6hw8" event={"ID":"973eb9f4-c418-4ca6-9cb1-bd5b4980b83f","Type":"ContainerStarted","Data":"6db6d4306939e8ab86976292ef459847f207faa37b96fd33860043f379746bdb"}
Oct 30 19:06:11.344750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:11.344725    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dcpr" event={"ID":"5250b4fe-4b08-4a7a-bfc7-d7343f202585","Type":"ContainerStarted","Data":"9df211b9dc9ffb2dbfe9de0317c32f4685bfafdd2a24c783ffff34d1b4298128"}
Oct 30 19:06:11.345985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:11.345952    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-882bp" event={"ID":"0413837b-14e3-4465-8a24-1bca4f394e06","Type":"ContainerStarted","Data":"913b48ccc6e0454f111fb4e67d4e19d3312408ad43fe742add3acb3f6e1d6bfb"}
Oct 30 19:06:11.670046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:11.670010    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:11.871243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:11.871157    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:12.272682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:12.272651    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:12.351887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:12.351846    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-882bp" event={"ID":"0413837b-14e3-4465-8a24-1bca4f394e06","Type":"ContainerStarted","Data":"24b25169729eb99c07b215151107d51d7d3f9452e1c978a4008406964318b683"}
Oct 30 19:06:12.352260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:12.351906    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-882bp" event={"ID":"0413837b-14e3-4465-8a24-1bca4f394e06","Type":"ContainerStarted","Data":"5a3e16f19e9868ca1a61f3c1a11d48075cab049ecd1bd8d32e78df21dc5646e9"}
Oct 30 19:06:12.353655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:12.353634    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dcpr" event={"ID":"5250b4fe-4b08-4a7a-bfc7-d7343f202585","Type":"ContainerStarted","Data":"deaf1c450232b525a6fd014d64cfd4e0454c1609f73f1e7a3b3dee44a768903d"}
Oct 30 19:06:12.353766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:12.353675    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dcpr" event={"ID":"5250b4fe-4b08-4a7a-bfc7-d7343f202585","Type":"ContainerStarted","Data":"e5d9ebdbd4b920e59e1abf1d957b062000e181f54df43891be515784ad096586"}
Oct 30 19:06:12.353802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:12.353773    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:12.432979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:12.432943    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:12.589824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:12.589797    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:13.074029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:13.073993    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:13.232560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:13.232523    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:13.470640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:13.470591    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:13.800983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:13.800887    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:13.915492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:13.915459    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:13.915632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:13.915507    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:13.918064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:13.918039    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6996469c-158c-46af-9e06-a89a9f7137c0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-rmqbj\" (UID: \"6996469c-158c-46af-9e06-a89a9f7137c0\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:13.918198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:13.918051    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6e8d41-096c-4121-b9c9-a147a7ee010d-metrics-certs\") pod \"router-default-ddbc459d5-jmrh9\" (UID: \"8c6e8d41-096c-4121-b9c9-a147a7ee010d\") " pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:14.011662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.011620    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:14.016561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.016544    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:14.016613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.016592    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:14.016669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.016627    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:14.016711 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:14.016673    4925 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:14.016752 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:14.016742    4925 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls podName:d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:22.016728951 +0000 UTC m=+16.919741966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4lg8" (UID: "d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:06:14.017281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.017263    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"
Oct 30 19:06:14.019242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.019224    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"image-registry-bd99d66bc-r6hq2\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") " pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:14.019313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.019257    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"image-registry-775fb78d4b-vms7p\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") " pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:14.019369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.019330    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:14.020431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.020411    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:14.036334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.036308    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:14.176080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.176050    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:14.325971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.325254    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj"]
Oct 30 19:06:14.329144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.329111    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-775fb78d4b-vms7p"]
Oct 30 19:06:14.329313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.329219    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddbc459d5-jmrh9"]
Oct 30 19:06:14.334741 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:14.334640    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6e8d41_096c_4121_b9c9_a147a7ee010d.slice/crio-62e5d6172d4527db6889755700e51d98897c62c48e0504fe95afcdab3b9b5e97 WatchSource:0}: Error finding container 62e5d6172d4527db6889755700e51d98897c62c48e0504fe95afcdab3b9b5e97: Status 404 returned error can't find the container with id 62e5d6172d4527db6889755700e51d98897c62c48e0504fe95afcdab3b9b5e97
Oct 30 19:06:14.335903 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:14.335872    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6996469c_158c_46af_9e06_a89a9f7137c0.slice/crio-a4affed7ef9e0405aebfc245ae24597dcc639955cf56c6dfc599fd049fae013d WatchSource:0}: Error finding container a4affed7ef9e0405aebfc245ae24597dcc639955cf56c6dfc599fd049fae013d: Status 404 returned error can't find the container with id a4affed7ef9e0405aebfc245ae24597dcc639955cf56c6dfc599fd049fae013d
Oct 30 19:06:14.336619 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:14.336548    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9d8478_9a05_4d2c_9210_a08764728647.slice/crio-80be1cae97b41b47aad73543a4756fea28fadaa684f848f3150e7d4f88cb16fa WatchSource:0}: Error finding container 80be1cae97b41b47aad73543a4756fea28fadaa684f848f3150e7d4f88cb16fa: Status 404 returned error can't find the container with id 80be1cae97b41b47aad73543a4756fea28fadaa684f848f3150e7d4f88cb16fa
Oct 30 19:06:14.349108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.349078    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bd99d66bc-r6hq2"]
Oct 30 19:06:14.356413 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:14.355279    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08977e9_bc18_4071_8748_2c6d7bc94f80.slice/crio-744c0a465651f2848244b3b0dbc2502a07a88dcb8c70abccafd49f84e88fa9e7 WatchSource:0}: Error finding container 744c0a465651f2848244b3b0dbc2502a07a88dcb8c70abccafd49f84e88fa9e7: Status 404 returned error can't find the container with id 744c0a465651f2848244b3b0dbc2502a07a88dcb8c70abccafd49f84e88fa9e7
Oct 30 19:06:14.356783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.356761    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:14.362469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.362449    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2" event={"ID":"c08977e9-bc18-4071-8748-2c6d7bc94f80","Type":"ContainerStarted","Data":"744c0a465651f2848244b3b0dbc2502a07a88dcb8c70abccafd49f84e88fa9e7"}
Oct 30 19:06:14.363862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.363840    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj" event={"ID":"6996469c-158c-46af-9e06-a89a9f7137c0","Type":"ContainerStarted","Data":"a4affed7ef9e0405aebfc245ae24597dcc639955cf56c6dfc599fd049fae013d"}
Oct 30 19:06:14.366250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.366228    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddbc459d5-jmrh9" event={"ID":"8c6e8d41-096c-4121-b9c9-a147a7ee010d","Type":"ContainerStarted","Data":"62e5d6172d4527db6889755700e51d98897c62c48e0504fe95afcdab3b9b5e97"}
Oct 30 19:06:14.367953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.367932    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p" event={"ID":"8b9d8478-9a05-4d2c-9210-a08764728647","Type":"ContainerStarted","Data":"80be1cae97b41b47aad73543a4756fea28fadaa684f848f3150e7d4f88cb16fa"}
Oct 30 19:06:14.676040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.676003    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:14.873130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:14.873036    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:15.373677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:15.373612    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2" event={"ID":"c08977e9-bc18-4071-8748-2c6d7bc94f80","Type":"ContainerStarted","Data":"c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23"}
Oct 30 19:06:15.375313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:15.375284    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddbc459d5-jmrh9" event={"ID":"8c6e8d41-096c-4121-b9c9-a147a7ee010d","Type":"ContainerStarted","Data":"28176f0d218a8498ed787c6ece7a174224dbfe0212b7492fcc2b151b8186e045"}
Oct 30 19:06:15.377128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:15.377102    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p" event={"ID":"8b9d8478-9a05-4d2c-9210-a08764728647","Type":"ContainerStarted","Data":"864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e"}
Oct 30 19:06:15.480012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:15.479945    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:15.679943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:15.679909    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:15.870336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:15.870302    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:15.890419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:15.890395    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:15.890829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:15.890817    4925 scope.go:122] "RemoveContainer" containerID="785b1dc18fe0bbc67e07f4d9d1da6815ba204f86fed8e0c4f931dbcb956f6d3d"
Oct 30 19:06:15.891019 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:15.890997    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fskzc_openshift-console-operator(664daa2d-471e-4380-bfdc-73df3a83ab3e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" podUID="664daa2d-471e-4380-bfdc-73df3a83ab3e"
Oct 30 19:06:16.019868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.019835    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:16.022632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.022614    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:16.060862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.060834    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:16.142079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.142039    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:16.341295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.341214    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:16.381895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.381872    4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:06:16.381895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.381872    4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:06:16.382070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.381867    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-rmqbj" event={"ID":"6996469c-158c-46af-9e06-a89a9f7137c0","Type":"ContainerStarted","Data":"09b8904755979c5ca3197183549d9c110dbaddd3c0fb92ed886a889585614c21"}
Oct 30 19:06:16.382070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.381930    4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:06:16.501420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.501387    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:16.659999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.659966    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:16.828287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:16.828250    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:17.200209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.200182    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:17.255103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.255079    4925 scope.go:122] "RemoveContainer" containerID="785b1dc18fe0bbc67e07f4d9d1da6815ba204f86fed8e0c4f931dbcb956f6d3d"
Oct 30 19:06:17.378468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.378435    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:17.387374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.387354    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:06:17.387513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.387481    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc" event={"ID":"664daa2d-471e-4380-bfdc-73df3a83ab3e","Type":"ContainerStarted","Data":"d1d4919dac928ba6c8a9c5930a6a673db1767c23b6b4bda355be868a30f9af8a"}
Oct 30 19:06:17.387939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.387923    4925 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:06:17.388030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.387962    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:17.534308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.534232    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:17.584302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.584278    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fskzc"
Oct 30 19:06:17.699915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.699875    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:17.701247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.701224    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-ddbc459d5-jmrh9"
Oct 30 19:06:17.900045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:17.900012    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:18.066192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:18.066155    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:18.224354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:18.224275    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:18.382891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:18.382855    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:18.636314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:18.636246    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgjj4"
Oct 30 19:06:18.800157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:18.800126    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:19.292858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:19.292829    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:19.392429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:19.392394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:19.558179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:19.558088    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:22.037337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.037294    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-775fb78d4b-vms7p"]
Oct 30 19:06:22.088994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.088965    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:22.091422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.091397    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4lg8\" (UID: \"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04\") " pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:22.139073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.139042    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4lg8"
Oct 30 19:06:22.197719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.196546    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:22.359438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.359354    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4dcpr"
Oct 30 19:06:22.383541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.383512    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:22.420070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.420043    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4lg8"]
Oct 30 19:06:22.425315 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:22.425279    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a4e9c9_0315_4f33_8ca6_2f5ca5d86d04.slice/crio-e5dcfc098bff5e5d14f9a7fdc712f4af59f655967c1c8478626e67ca8a553557 WatchSource:0}: Error finding container e5dcfc098bff5e5d14f9a7fdc712f4af59f655967c1c8478626e67ca8a553557: Status 404 returned error can't find the container with id e5dcfc098bff5e5d14f9a7fdc712f4af59f655967c1c8478626e67ca8a553557
Oct 30 19:06:22.566439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.566411    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:22.729792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:22.729756    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:23.410409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:23.410378    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4lg8" event={"ID":"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04","Type":"ContainerStarted","Data":"a6057045efceb4c591f8cebdd191bf4cd5b64b9eec6b759aefde66b717fdae01"}
Oct 30 19:06:23.410833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:23.410437    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4lg8" event={"ID":"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04","Type":"ContainerStarted","Data":"8e33de2e23fdad639f74cbd16e9acd804bc999a37872cd92bb9976fd7fe48ddd"}
Oct 30 19:06:23.410833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:23.410459    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4lg8" event={"ID":"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04","Type":"ContainerStarted","Data":"e5dcfc098bff5e5d14f9a7fdc712f4af59f655967c1c8478626e67ca8a553557"}
Oct 30 19:06:23.433614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:23.433582    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dcpr_5250b4fe-4b08-4a7a-bfc7-d7343f202585/dns/0.log"
Oct 30 19:06:23.632882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:23.632811    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dcpr_5250b4fe-4b08-4a7a-bfc7-d7343f202585/kube-rbac-proxy/0.log"
Oct 30 19:06:24.632599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:24.632573    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g76xc_a50accc7-ce62-4f71-9a4a-aa128cfe10c4/dns-node-resolver/0.log"
Oct 30 19:06:25.033446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:25.033421    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-775fb78d4b-vms7p_8b9d8478-9a05-4d2c-9210-a08764728647/registry/0.log"
Oct 30 19:06:25.233284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:25.233235    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-bd99d66bc-r6hq2_c08977e9-bc18-4071-8748-2c6d7bc94f80/registry/0.log"
Oct 30 19:06:25.420526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:25.420491    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4lg8" event={"ID":"d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04","Type":"ContainerStarted","Data":"c81fb58a1dee20bcecc044382ebceb783f3163cf86cf36f727bddb188c4632f0"}
Oct 30 19:06:25.832859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:25.832785    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cnsll_9a69fe59-2039-43ff-a132-eeed44422412/node-ca/0.log"
Oct 30 19:06:26.033589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:26.033556    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddbc459d5-jmrh9_8c6e8d41-096c-4121-b9c9-a147a7ee010d/router/0.log"
Oct 30 19:06:26.233356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:26.233329    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nb5x6_a09e9550-86c1-474b-9774-50dd9189f3fa/serve-healthcheck-canary/0.log"
Oct 30 19:06:26.606137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:26.606059    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:26.807558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:26.807525    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:26.965221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:26.965191    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:27.128019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:27.127994    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-24ddg"
Oct 30 19:06:28.824107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:28.824072    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:30.410910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:30.410875    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:30.600962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:30.600932    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:31.832550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:31.832512    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:32.009496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.009414    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:32.043996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.043968    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:32.130510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.130477    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-25vxx"]
Oct 30 19:06:32.162387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.162362    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.169560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.169516    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:32.204822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.204794    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:06:32.204822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.204819    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:06:32.205024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.205005    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:06:32.205070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.205060    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:06:32.205202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.205185    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n985b\""
Oct 30 19:06:32.268746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.268660    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.268746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.268724    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-textfile\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.268922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.268820    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f962e9e6-b6f6-40c5-a903-d44e902f40b9-metrics-client-ca\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.268922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.268845    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v94x\" (UniqueName: \"kubernetes.io/projected/f962e9e6-b6f6-40c5-a903-d44e902f40b9-kube-api-access-4v94x\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.268922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.268869    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-tls\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.268922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.268893    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.269092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.268937    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-wtmp\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.269092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.268972    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f962e9e6-b6f6-40c5-a903-d44e902f40b9-root\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.269092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.269072    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f962e9e6-b6f6-40c5-a903-d44e902f40b9-sys\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.335973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.335939    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:32.370096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370064    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f962e9e6-b6f6-40c5-a903-d44e902f40b9-sys\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370100    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370125    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-textfile\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370143    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f962e9e6-b6f6-40c5-a903-d44e902f40b9-metrics-client-ca\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370158    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v94x\" (UniqueName: \"kubernetes.io/projected/f962e9e6-b6f6-40c5-a903-d44e902f40b9-kube-api-access-4v94x\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370172    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f962e9e6-b6f6-40c5-a903-d44e902f40b9-sys\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370182    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-tls\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370254    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370282    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-wtmp\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370310    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f962e9e6-b6f6-40c5-a903-d44e902f40b9-root\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370397    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f962e9e6-b6f6-40c5-a903-d44e902f40b9-root\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370501    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-wtmp\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370600    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-textfile\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370911    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f962e9e6-b6f6-40c5-a903-d44e902f40b9-metrics-client-ca\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.370998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.370919    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.372802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.372784    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.372929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.372868    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f962e9e6-b6f6-40c5-a903-d44e902f40b9-node-exporter-tls\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.381704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.381670    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v94x\" (UniqueName: \"kubernetes.io/projected/f962e9e6-b6f6-40c5-a903-d44e902f40b9-kube-api-access-4v94x\") pod \"node-exporter-25vxx\" (UID: \"f962e9e6-b6f6-40c5-a903-d44e902f40b9\") " pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.512583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.512541    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-25vxx"
Oct 30 19:06:32.549288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.549263    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:32.552228 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:06:32.552205    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf962e9e6_b6f6_40c5_a903_d44e902f40b9.slice/crio-dc7d31ed4801e45c4f1c3662aa1c41ee2dba5383728db035e0ef89c8ecf9a6aa WatchSource:0}: Error finding container dc7d31ed4801e45c4f1c3662aa1c41ee2dba5383728db035e0ef89c8ecf9a6aa: Status 404 returned error can't find the container with id dc7d31ed4801e45c4f1c3662aa1c41ee2dba5383728db035e0ef89c8ecf9a6aa
Oct 30 19:06:32.709349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.709321    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:32.866328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:32.866242    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:33.046734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:33.046681    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:33.245388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:33.245356    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:33.457241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:33.457206    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25vxx" event={"ID":"f962e9e6-b6f6-40c5-a903-d44e902f40b9","Type":"ContainerStarted","Data":"dc7d31ed4801e45c4f1c3662aa1c41ee2dba5383728db035e0ef89c8ecf9a6aa"}
Oct 30 19:06:34.463530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:34.463498    4925 generic.go:356] "Generic (PLEG): container finished" podID="f962e9e6-b6f6-40c5-a903-d44e902f40b9" containerID="bd99a63a507bbc48d3184f4c16e180c37e1b3a2dd470be6eafb848d20b67deaa" exitCode=0
Oct 30 19:06:34.463952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:34.463584    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25vxx" event={"ID":"f962e9e6-b6f6-40c5-a903-d44e902f40b9","Type":"ContainerDied","Data":"bd99a63a507bbc48d3184f4c16e180c37e1b3a2dd470be6eafb848d20b67deaa"}
Oct 30 19:06:35.196850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:35.196820    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:35.399603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:35.399570    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:35.469365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:35.469286    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25vxx" event={"ID":"f962e9e6-b6f6-40c5-a903-d44e902f40b9","Type":"ContainerStarted","Data":"af7431a609a350df10fb88c488ac69e028eddeb9581d8a14060c09cd9a479479"}
Oct 30 19:06:35.469365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:35.469347    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25vxx" event={"ID":"f962e9e6-b6f6-40c5-a903-d44e902f40b9","Type":"ContainerStarted","Data":"8065c55be5201884a6eeac1280439fddcd92dcc5d5dede5e75ba63820dc67787"}
Oct 30 19:06:35.491982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:35.491921    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-25vxx" podStartSLOduration=1.946174549 podStartE2EDuration="3.4919091s" totalImagesPullingTime="1.461814682s" totalInitContainerRuntime="83.919869ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:35.490933897 +0000 UTC m=+30.393946931" watchObservedRunningTime="2026-10-30 19:06:35.4919091 +0000 UTC m=+30.394922136"
Oct 30 19:06:35.575904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:35.575874    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:36.878183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:36.878150    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:37.086237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:37.086191    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:37.392159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:37.392127    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:06:37.817748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:37.817663    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:38.020681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:38.020641    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:38.251947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:38.251912    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9s7mb"
Oct 30 19:06:38.468279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:38.468250    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:41.153108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:41.153079    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:41.329086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:41.329046    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:41.529660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:41.529586    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:41.696275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:41.696239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:45.593411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:45.593379    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bd99d66bc-r6hq2"]
Oct 30 19:06:45.847352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:45.847264    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:46.008427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:46.008351    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:47.060117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.060054    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p" podUID="8b9d8478-9a05-4d2c-9210-a08764728647" containerName="registry" containerID="cri-o://864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e" gracePeriod=30
Oct 30 19:06:47.296595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.296569    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:47.400771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.400745    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967\") pod \"8b9d8478-9a05-4d2c-9210-a08764728647\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") "
Oct 30 19:06:47.400943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.400779    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token\") pod \"8b9d8478-9a05-4d2c-9210-a08764728647\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") "
Oct 30 19:06:47.400943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.400810    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted\") pod \"8b9d8478-9a05-4d2c-9210-a08764728647\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") "
Oct 30 19:06:47.400943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.400922    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") pod \"8b9d8478-9a05-4d2c-9210-a08764728647\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") "
Oct 30 19:06:47.401111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.400976    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca\") pod \"8b9d8478-9a05-4d2c-9210-a08764728647\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") "
Oct 30 19:06:47.401111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.401014    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets\") pod \"8b9d8478-9a05-4d2c-9210-a08764728647\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") "
Oct 30 19:06:47.401111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.401040    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration\") pod \"8b9d8478-9a05-4d2c-9210-a08764728647\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") "
Oct 30 19:06:47.401111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.401089    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates\") pod \"8b9d8478-9a05-4d2c-9210-a08764728647\" (UID: \"8b9d8478-9a05-4d2c-9210-a08764728647\") "
Oct 30 19:06:47.401729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.401454    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca" pod "8b9d8478-9a05-4d2c-9210-a08764728647" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:47.401729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.401628    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates" pod "8b9d8478-9a05-4d2c-9210-a08764728647" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:47.403487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.403465    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token" pod "8b9d8478-9a05-4d2c-9210-a08764728647" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:47.403593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.403566    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets" pod "8b9d8478-9a05-4d2c-9210-a08764728647" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:47.403735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.403712    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls" pod "8b9d8478-9a05-4d2c-9210-a08764728647" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:47.403823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.403807    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967" pod "8b9d8478-9a05-4d2c-9210-a08764728647" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647"). InnerVolumeSpecName "kube-api-access-rk967". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:47.404033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.404011    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration" pod "8b9d8478-9a05-4d2c-9210-a08764728647" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:47.407939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.407919    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted" pod "8b9d8478-9a05-4d2c-9210-a08764728647" (UID: "8b9d8478-9a05-4d2c-9210-a08764728647"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:47.502026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.502005    4925 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-installation-pull-secrets\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:47.502026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.502025    4925 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8b9d8478-9a05-4d2c-9210-a08764728647-image-registry-private-configuration\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:47.502156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.502035    4925 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-registry-certificates\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:47.502156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.502044    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk967\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-kube-api-access-rk967\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:47.502156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.502053    4925 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-bound-sa-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:47.502156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.502062    4925 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8b9d8478-9a05-4d2c-9210-a08764728647-ca-trust-extracted\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:47.502156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.502071    4925 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8b9d8478-9a05-4d2c-9210-a08764728647-registry-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:47.502156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.502079    4925 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8b9d8478-9a05-4d2c-9210-a08764728647-trusted-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:47.515607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.515583    4925 generic.go:356] "Generic (PLEG): container finished" podID="8b9d8478-9a05-4d2c-9210-a08764728647" containerID="864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e" exitCode=0
Oct 30 19:06:47.515731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.515630    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p"
Oct 30 19:06:47.515731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.515639    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p" event={"ID":"8b9d8478-9a05-4d2c-9210-a08764728647","Type":"ContainerDied","Data":"864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e"}
Oct 30 19:06:47.515731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.515722    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-775fb78d4b-vms7p" event={"ID":"8b9d8478-9a05-4d2c-9210-a08764728647","Type":"ContainerDied","Data":"80be1cae97b41b47aad73543a4756fea28fadaa684f848f3150e7d4f88cb16fa"}
Oct 30 19:06:47.515900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.515751    4925 scope.go:122] "RemoveContainer" containerID="864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e"
Oct 30 19:06:47.524616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.524595    4925 scope.go:122] "RemoveContainer" containerID="864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e"
Oct 30 19:06:47.524911 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:47.524893    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e\": container with ID starting with 864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e not found: ID does not exist" containerID="864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e"
Oct 30 19:06:47.524960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.524920    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e"} err="failed to get container status \"864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e\": rpc error: code = NotFound desc = could not find container \"864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e\": container with ID starting with 864dc6d2562022b5850b0033a7770cb8499891c65fbf42190b0f70340ddf5b9e not found: ID does not exist"
Oct 30 19:06:47.536210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.536185    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-775fb78d4b-vms7p"]
Oct 30 19:06:47.549116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:47.549094    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-775fb78d4b-vms7p"]
Oct 30 19:06:48.338069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:48.338038    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:48.504167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:48.504133    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:49.260628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:49.260597    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9d8478-9a05-4d2c-9210-a08764728647" path="/var/lib/kubelet/pods/8b9d8478-9a05-4d2c-9210-a08764728647/volumes"
Oct 30 19:06:49.816901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:49.816867    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:49.979710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:49.979658    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:52.124001 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:06:52.123963    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:06:54.842398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:54.842361    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:55.046597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:55.046559    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:55.741491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:55.741452    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:55.917772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:55.917728    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:58.507754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:58.507723    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:58.669732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:06:58.669657    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:02.380042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:02.380001    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:02.541752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:02.541722    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:10.609382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.609349    4925 generic.go:356] "Generic (PLEG): container finished" podID="40c4bb90-1b90-4e9c-9822-5daafa73cdfa" containerID="9d3f51aea9a11ec6df5cc6fb0defd685ffabf9ac37ee993b34faca222a63d3d4" exitCode=0
Oct 30 19:07:10.609826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.609433    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7" event={"ID":"40c4bb90-1b90-4e9c-9822-5daafa73cdfa","Type":"ContainerDied","Data":"9d3f51aea9a11ec6df5cc6fb0defd685ffabf9ac37ee993b34faca222a63d3d4"}
Oct 30 19:07:10.609826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.609787    4925 scope.go:122] "RemoveContainer" containerID="9d3f51aea9a11ec6df5cc6fb0defd685ffabf9ac37ee993b34faca222a63d3d4"
Oct 30 19:07:10.614206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.614176    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2" podUID="c08977e9-bc18-4071-8748-2c6d7bc94f80" containerName="registry" containerID="cri-o://c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23" gracePeriod=30
Oct 30 19:07:10.857679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.857656    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:07:10.970512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.970481    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets\") pod \"c08977e9-bc18-4071-8748-2c6d7bc94f80\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") "
Oct 30 19:07:10.970716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.970528    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr\") pod \"c08977e9-bc18-4071-8748-2c6d7bc94f80\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") "
Oct 30 19:07:10.970716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.970579    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") pod \"c08977e9-bc18-4071-8748-2c6d7bc94f80\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") "
Oct 30 19:07:10.970716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.970605    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration\") pod \"c08977e9-bc18-4071-8748-2c6d7bc94f80\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") "
Oct 30 19:07:10.970716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.970640    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted\") pod \"c08977e9-bc18-4071-8748-2c6d7bc94f80\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") "
Oct 30 19:07:10.970931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.970800    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca\") pod \"c08977e9-bc18-4071-8748-2c6d7bc94f80\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") "
Oct 30 19:07:10.971210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.971177    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca" pod "c08977e9-bc18-4071-8748-2c6d7bc94f80" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:10.971741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.971453    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token\") pod \"c08977e9-bc18-4071-8748-2c6d7bc94f80\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") "
Oct 30 19:07:10.971741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.971524    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates\") pod \"c08977e9-bc18-4071-8748-2c6d7bc94f80\" (UID: \"c08977e9-bc18-4071-8748-2c6d7bc94f80\") "
Oct 30 19:07:10.971889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.971763    4925 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-trusted-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:10.972234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.972211    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates" pod "c08977e9-bc18-4071-8748-2c6d7bc94f80" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:10.973270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.973241    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr" pod "c08977e9-bc18-4071-8748-2c6d7bc94f80" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80"). InnerVolumeSpecName "kube-api-access-mnrvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:10.973540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.973518    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets" pod "c08977e9-bc18-4071-8748-2c6d7bc94f80" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:10.973605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.973578    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration" pod "c08977e9-bc18-4071-8748-2c6d7bc94f80" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:10.973645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.973601    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls" pod "c08977e9-bc18-4071-8748-2c6d7bc94f80" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:10.974121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.974097    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token" pod "c08977e9-bc18-4071-8748-2c6d7bc94f80" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:10.977840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:10.977820    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted" pod "c08977e9-bc18-4071-8748-2c6d7bc94f80" (UID: "c08977e9-bc18-4071-8748-2c6d7bc94f80"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:11.072671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.072637    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnrvr\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-kube-api-access-mnrvr\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:11.072671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.072665    4925 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:11.072671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.072676    4925 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-image-registry-private-configuration\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:11.072900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.072704    4925 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c08977e9-bc18-4071-8748-2c6d7bc94f80-ca-trust-extracted\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:11.072900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.072716    4925 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c08977e9-bc18-4071-8748-2c6d7bc94f80-bound-sa-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:11.072900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.072725    4925 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c08977e9-bc18-4071-8748-2c6d7bc94f80-registry-certificates\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:11.072900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.072734    4925 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c08977e9-bc18-4071-8748-2c6d7bc94f80-installation-pull-secrets\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:11.477975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.477941    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:11.614772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.614677    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-stxf7" event={"ID":"40c4bb90-1b90-4e9c-9822-5daafa73cdfa","Type":"ContainerStarted","Data":"c65ac64a7cbe3cbd73a4a7aa8ba32eae3b8890144fcd0543937e51de2667dcdd"}
Oct 30 19:07:11.615988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.615964    4925 generic.go:356] "Generic (PLEG): container finished" podID="c08977e9-bc18-4071-8748-2c6d7bc94f80" containerID="c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23" exitCode=0
Oct 30 19:07:11.616107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.616003    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2" event={"ID":"c08977e9-bc18-4071-8748-2c6d7bc94f80","Type":"ContainerDied","Data":"c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23"}
Oct 30 19:07:11.616107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.616032    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2" event={"ID":"c08977e9-bc18-4071-8748-2c6d7bc94f80","Type":"ContainerDied","Data":"744c0a465651f2848244b3b0dbc2502a07a88dcb8c70abccafd49f84e88fa9e7"}
Oct 30 19:07:11.616107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.616051    4925 scope.go:122] "RemoveContainer" containerID="c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23"
Oct 30 19:07:11.616107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.616003    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bd99d66bc-r6hq2"
Oct 30 19:07:11.624156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.624137    4925 scope.go:122] "RemoveContainer" containerID="c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23"
Oct 30 19:07:11.624439 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:07:11.624421    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23\": container with ID starting with c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23 not found: ID does not exist" containerID="c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23"
Oct 30 19:07:11.624480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.624446    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23"} err="failed to get container status \"c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23\": rpc error: code = NotFound desc = could not find container \"c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23\": container with ID starting with c09b9a6ed2695f3c887886f4c8bdbdf9cafb31492f65136fdc42634dcc8f5b23 not found: ID does not exist"
Oct 30 19:07:11.648365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.648343    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bd99d66bc-r6hq2"]
Oct 30 19:07:11.650966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.650946    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bd99d66bc-r6hq2"]
Oct 30 19:07:11.731224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:11.731197    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:13.259496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:13.259463    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08977e9-bc18-4071-8748-2c6d7bc94f80" path="/var/lib/kubelet/pods/c08977e9-bc18-4071-8748-2c6d7bc94f80/volumes"
Oct 30 19:07:22.047279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:22.047239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:22.123700 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:07:22.123646    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:07:22.221084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:22.221050    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:22.628866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:22.628830    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:35.707733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:35.707676    4925 generic.go:356] "Generic (PLEG): container finished" podID="151fec2f-192b-4af4-9d4e-10a790100cb9" containerID="b3f2e00bcbcb949c9feafd22168cc3abce728a37aa0bec6383e883aaee46fcc2" exitCode=0
Oct 30 19:07:35.708185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:35.707750    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j" event={"ID":"151fec2f-192b-4af4-9d4e-10a790100cb9","Type":"ContainerDied","Data":"b3f2e00bcbcb949c9feafd22168cc3abce728a37aa0bec6383e883aaee46fcc2"}
Oct 30 19:07:35.708185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:35.708060    4925 scope.go:122] "RemoveContainer" containerID="b3f2e00bcbcb949c9feafd22168cc3abce728a37aa0bec6383e883aaee46fcc2"
Oct 30 19:07:36.712707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:36.712655    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hft8j" event={"ID":"151fec2f-192b-4af4-9d4e-10a790100cb9","Type":"ContainerStarted","Data":"558b198b2abd07117261b7a907d472021ca63cc7922425e7f2336ec22c0a8d60"}
Oct 30 19:07:50.362109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:50.362072    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:50.529291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:50.529258    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:52.123664 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:07:52.123621    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:07:54.924507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:07:54.924459    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:06.876528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:06.876484    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:07.044175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:07.044145    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:10.861593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:10.861560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:18.062387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:18.062292    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:18.237981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:18.237947    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:19.489320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:19.489287    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:19.647426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:19.647386    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:20.100642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:20.100605    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:22.124528 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:08:22.124495    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:08:23.428316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:23.428279    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:32.384184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:32.384143    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:32.558468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:32.558430    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:52.124423 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:08:52.124390    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:08:54.968643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:08:54.968590    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:22.123756 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:09:22.123719    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:09:26.797476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:09:26.797439    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:52.123613 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:09:52.123527    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:09:55.031444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:09:55.031411    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:55.403558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:09:55.403522    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:20.945634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:20.945596    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:22.124130 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:10:22.124095    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:10:22.450143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:22.450095    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:43.205536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:43.205495    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:44.644873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:44.644820    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:46.746292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:46.746255    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:46.944557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:46.944520    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:48.881079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:48.881048    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:49.076534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:49.076501    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:49.286830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:49.286792    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:51.264459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.264427    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:51.501593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.501556    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:51.957712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.957663    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"]
Oct 30 19:10:51.958043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.958026    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8b9d8478-9a05-4d2c-9210-a08764728647" containerName="registry"
Oct 30 19:10:51.958093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.958045    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9d8478-9a05-4d2c-9210-a08764728647" containerName="registry"
Oct 30 19:10:51.958093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.958056    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c08977e9-bc18-4071-8748-2c6d7bc94f80" containerName="registry"
Oct 30 19:10:51.958093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.958061    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08977e9-bc18-4071-8748-2c6d7bc94f80" containerName="registry"
Oct 30 19:10:51.958180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.958113    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9d8478-9a05-4d2c-9210-a08764728647" containerName="registry"
Oct 30 19:10:51.958180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.958121    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08977e9-bc18-4071-8748-2c6d7bc94f80" containerName="registry"
Oct 30 19:10:51.960801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.960786    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"
Oct 30 19:10:51.963943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.963919    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 19:10:51.964143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.964129    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 19:10:51.964202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.964145    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-v5ntp\""
Oct 30 19:10:51.969292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:51.969271    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"]
Oct 30 19:10:52.067096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:52.067072    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67ac240e-8f17-4eb9-a0f5-665a113d4985-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dgvwp\" (UID: \"67ac240e-8f17-4eb9-a0f5-665a113d4985\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"
Oct 30 19:10:52.067217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:52.067124    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gk6r\" (UniqueName: \"kubernetes.io/projected/67ac240e-8f17-4eb9-a0f5-665a113d4985-kube-api-access-5gk6r\") pod \"cert-manager-cainjector-5c57bbd4fc-dgvwp\" (UID: \"67ac240e-8f17-4eb9-a0f5-665a113d4985\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"
Oct 30 19:10:52.124255 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:10:52.124224    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:10:52.168476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:52.168447    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67ac240e-8f17-4eb9-a0f5-665a113d4985-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dgvwp\" (UID: \"67ac240e-8f17-4eb9-a0f5-665a113d4985\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"
Oct 30 19:10:52.168617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:52.168515    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gk6r\" (UniqueName: \"kubernetes.io/projected/67ac240e-8f17-4eb9-a0f5-665a113d4985-kube-api-access-5gk6r\") pod \"cert-manager-cainjector-5c57bbd4fc-dgvwp\" (UID: \"67ac240e-8f17-4eb9-a0f5-665a113d4985\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"
Oct 30 19:10:52.180207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:52.180181    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67ac240e-8f17-4eb9-a0f5-665a113d4985-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dgvwp\" (UID: \"67ac240e-8f17-4eb9-a0f5-665a113d4985\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"
Oct 30 19:10:52.180313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:52.180297    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gk6r\" (UniqueName: \"kubernetes.io/projected/67ac240e-8f17-4eb9-a0f5-665a113d4985-kube-api-access-5gk6r\") pod \"cert-manager-cainjector-5c57bbd4fc-dgvwp\" (UID: \"67ac240e-8f17-4eb9-a0f5-665a113d4985\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"
Oct 30 19:10:52.282669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:52.282612    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"
Oct 30 19:10:52.554593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:52.554518    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp"]
Oct 30 19:10:52.558425 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:10:52.558396    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67ac240e_8f17_4eb9_a0f5_665a113d4985.slice/crio-17006c9158ffd35d9dbeb3af480e6f208506ec74afcfca2aa83593b364f1bf68 WatchSource:0}: Error finding container 17006c9158ffd35d9dbeb3af480e6f208506ec74afcfca2aa83593b364f1bf68: Status 404 returned error can't find the container with id 17006c9158ffd35d9dbeb3af480e6f208506ec74afcfca2aa83593b364f1bf68
Oct 30 19:10:53.409059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:53.409014    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp" event={"ID":"67ac240e-8f17-4eb9-a0f5-665a113d4985","Type":"ContainerStarted","Data":"17006c9158ffd35d9dbeb3af480e6f208506ec74afcfca2aa83593b364f1bf68"}
Oct 30 19:10:53.605310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:53.605278    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:53.773602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:53.773513    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:55.092619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:55.092583    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:56.423344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:56.423301    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp" event={"ID":"67ac240e-8f17-4eb9-a0f5-665a113d4985","Type":"ContainerStarted","Data":"2a2a50d5532e084b6cb442a531dc62f2ab4c73148b9df0273d5fa30248979f97"}
Oct 30 19:10:56.440806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:56.440759    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dgvwp" podStartSLOduration=2.231088889 podStartE2EDuration="5.440745987s" totalImagesPullingTime="3.209657098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:56.439753515 +0000 UTC m=+291.342766555" watchObservedRunningTime="2026-10-30 19:10:56.440745987 +0000 UTC m=+291.343759023"
Oct 30 19:10:56.947395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:56.947353    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:57.107075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:57.107042    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:58.369305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:58.369273    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:58.523419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:10:58.523385    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:00.807104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:00.807071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:00.960224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:00.960190    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:05.271361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:05.271332    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:11:05.272806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:05.272784    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:11:19.664642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:19.664551    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:19.834317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:19.834273    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:21.329545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:21.329513    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:22.124758 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:11:22.124725    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:11:25.578975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.578938    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"]
Oct 30 19:11:25.582446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.582426    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.590563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.590537    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:25.593699 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.593660    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"]
Oct 30 19:11:25.609597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.609569    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-cert\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.609700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.609599    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-manager-config\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.609700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.609622    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-metrics-cert\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.609700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.609637    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrcl8\" (UniqueName: \"kubernetes.io/projected/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-kube-api-access-qrcl8\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.684069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.684032    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 19:11:25.684253 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.684225    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 19:11:25.684349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.684332    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:11:25.684676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.684664    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-djrzj\""
Oct 30 19:11:25.684762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.684675    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 19:11:25.684762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.684681    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 19:11:25.710471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.710448    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-cert\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.710576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.710480    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-manager-config\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.710576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.710505    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-metrics-cert\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.710576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.710521    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrcl8\" (UniqueName: \"kubernetes.io/projected/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-kube-api-access-qrcl8\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.711154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.711129    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-manager-config\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.713194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.713170    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-metrics-cert\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.713341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.713320    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-cert\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.721791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.721771    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrcl8\" (UniqueName: \"kubernetes.io/projected/4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c-kube-api-access-qrcl8\") pod \"lws-controller-manager-8c744547b-fv4mb\" (UID: \"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c\") " pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:25.780657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.780630    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:25.893943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:25.893925    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:26.169285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:26.169214    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"]
Oct 30 19:11:26.173259 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:11:26.173227    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbe72f0_8cc8_4da0_927c_8a1e23cdeb4c.slice/crio-199d7a0a47046489d48d822f5f5be4e79edda206ea2f12f1f1a5d789d29e2980 WatchSource:0}: Error finding container 199d7a0a47046489d48d822f5f5be4e79edda206ea2f12f1f1a5d789d29e2980: Status 404 returned error can't find the container with id 199d7a0a47046489d48d822f5f5be4e79edda206ea2f12f1f1a5d789d29e2980
Oct 30 19:11:26.175020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:26.175005    4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:11:26.544515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:26.544430    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb" event={"ID":"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c","Type":"ContainerStarted","Data":"199d7a0a47046489d48d822f5f5be4e79edda206ea2f12f1f1a5d789d29e2980"}
Oct 30 19:11:29.558075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:29.558039    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb" event={"ID":"4cbe72f0-8cc8-4da0-927c-8a1e23cdeb4c","Type":"ContainerStarted","Data":"81dfdbf36140392de5204d0c1d0758de4874864e47d3679206ac792d37825aa2"}
Oct 30 19:11:29.558505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:29.558175    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:29.577545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:29.577495    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb" podStartSLOduration=1.8007508209999998 podStartE2EDuration="4.57747886s" totalImagesPullingTime="2.776728039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:29.576897155 +0000 UTC m=+324.479910192" watchObservedRunningTime="2026-10-30 19:11:29.57747886 +0000 UTC m=+324.480491898"
Oct 30 19:11:36.213043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:36.213005    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:36.390517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:36.390473    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:37.026276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.026244    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:37.174325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.174285    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"]
Oct 30 19:11:37.178064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.178042    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.201254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.201232    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b74ba3b2-3630-4bd7-ab04-674a8795958e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.201358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.201260    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8rn\" (UniqueName: \"kubernetes.io/projected/b74ba3b2-3630-4bd7-ab04-674a8795958e-kube-api-access-kb8rn\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.201358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.201290    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b74ba3b2-3630-4bd7-ab04-674a8795958e-tmp\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.201443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.201385    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b74ba3b2-3630-4bd7-ab04-674a8795958e-manifests\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.201443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.201429    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b74ba3b2-3630-4bd7-ab04-674a8795958e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.201508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.201444    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b74ba3b2-3630-4bd7-ab04-674a8795958e-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.207490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.207467    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"]
Oct 30 19:11:37.248111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.248083    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:11:37.248432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.248113    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:11:37.248432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.248168    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:11:37.248432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.248353    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:11:37.248547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.248494    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:11:37.248547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.248498    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v5t6d\""
Oct 30 19:11:37.302128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.302058    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b74ba3b2-3630-4bd7-ab04-674a8795958e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.302128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.302099    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8rn\" (UniqueName: \"kubernetes.io/projected/b74ba3b2-3630-4bd7-ab04-674a8795958e-kube-api-access-kb8rn\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.302311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.302146    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b74ba3b2-3630-4bd7-ab04-674a8795958e-tmp\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.302311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.302192    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b74ba3b2-3630-4bd7-ab04-674a8795958e-manifests\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.302311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.302239    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b74ba3b2-3630-4bd7-ab04-674a8795958e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.302311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.302257    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b74ba3b2-3630-4bd7-ab04-674a8795958e-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.302644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.302617    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b74ba3b2-3630-4bd7-ab04-674a8795958e-manifests\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.304703 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.304670    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b74ba3b2-3630-4bd7-ab04-674a8795958e-tmp\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.304851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.304832    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b74ba3b2-3630-4bd7-ab04-674a8795958e-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.304969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.304951    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b74ba3b2-3630-4bd7-ab04-674a8795958e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.305045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.305026    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b74ba3b2-3630-4bd7-ab04-674a8795958e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.314395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.314372    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8rn\" (UniqueName: \"kubernetes.io/projected/b74ba3b2-3630-4bd7-ab04-674a8795958e-kube-api-access-kb8rn\") pod \"opendatahub-operator-controller-manager-7c94c66984-tv7w5\" (UID: \"b74ba3b2-3630-4bd7-ab04-674a8795958e\") " pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.358766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.358731    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:37.492602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.492580    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:37.548089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.548032    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:37.733749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.733707    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:37.794175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:37.794142    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"]
Oct 30 19:11:37.799646 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:11:37.799621    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74ba3b2_3630_4bd7_ab04_674a8795958e.slice/crio-f9770979742356d71c5f15f90e4b8170a7e94cd2dbd34d83b11518110192ff6e WatchSource:0}: Error finding container f9770979742356d71c5f15f90e4b8170a7e94cd2dbd34d83b11518110192ff6e: Status 404 returned error can't find the container with id f9770979742356d71c5f15f90e4b8170a7e94cd2dbd34d83b11518110192ff6e
Oct 30 19:11:38.307755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:38.307711    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:38.470190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:38.470149    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:38.597925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:38.597845    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5" event={"ID":"b74ba3b2-3630-4bd7-ab04-674a8795958e","Type":"ContainerStarted","Data":"f9770979742356d71c5f15f90e4b8170a7e94cd2dbd34d83b11518110192ff6e"}
Oct 30 19:11:40.564223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:40.564197    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8c744547b-fv4mb"
Oct 30 19:11:41.612189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:41.612150    4925 generic.go:356] "Generic (PLEG): container finished" podID="b74ba3b2-3630-4bd7-ab04-674a8795958e" containerID="dff700fbe36e37edda6e7a47fac0249a1601cf305e5fc43db6f5fa7cb49daa2c" exitCode=0
Oct 30 19:11:41.612570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:41.612236    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5" event={"ID":"b74ba3b2-3630-4bd7-ab04-674a8795958e","Type":"ContainerDied","Data":"dff700fbe36e37edda6e7a47fac0249a1601cf305e5fc43db6f5fa7cb49daa2c"}
Oct 30 19:11:42.618304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:42.618270    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5" event={"ID":"b74ba3b2-3630-4bd7-ab04-674a8795958e","Type":"ContainerStarted","Data":"0b73683eecccc9e0d1960939640317aded903389155058db4f2e6433584d5f39"}
Oct 30 19:11:42.618751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:42.618383    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:42.661114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:42.661073    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5" podStartSLOduration=2.024261257 podStartE2EDuration="5.661059814s" totalImagesPullingTime="3.20416955s" totalInitContainerRuntime="432.629007ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:42.659819876 +0000 UTC m=+337.562832913" watchObservedRunningTime="2026-10-30 19:11:42.661059814 +0000 UTC m=+337.564072842"
Oct 30 19:11:52.124704 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:11:52.124651    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:11:53.624495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:53.624466    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c94c66984-tv7w5"
Oct 30 19:11:55.158165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:55.158117    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:55.897715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:55.897647    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:56.138523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:56.138493    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:56.305224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:56.305143    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:56.480097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:56.480063    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:57.434981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:57.434947    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:57.716948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:57.716916    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:57.972557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:57.972478    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:58.128322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:58.128281    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:59.233437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:11:59.233397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:02.258337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:02.258293    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:02.421486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:02.421444    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:03.217490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.217446    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"]
Oct 30 19:12:03.223971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.223946    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.226980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.226959    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:12:03.226980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.226952    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xtp54\""
Oct 30 19:12:03.232439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.232414    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"]
Oct 30 19:12:03.312632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.312596    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.312632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.312634    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83805502-d8df-466d-a7c3-c842d05347d2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.313140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.312715    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.313140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.312753    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83805502-d8df-466d-a7c3-c842d05347d2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.313140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.312783    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdlvp\" (UniqueName: \"kubernetes.io/projected/83805502-d8df-466d-a7c3-c842d05347d2-kube-api-access-zdlvp\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.313140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.312806    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.313140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.312822    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.313140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.312926    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83805502-d8df-466d-a7c3-c842d05347d2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.313140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.313028    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.329787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.329751    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:03.413998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.413962    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83805502-d8df-466d-a7c3-c842d05347d2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414009    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdlvp\" (UniqueName: \"kubernetes.io/projected/83805502-d8df-466d-a7c3-c842d05347d2-kube-api-access-zdlvp\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414040    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414066    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414093    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83805502-d8df-466d-a7c3-c842d05347d2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414122    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414171    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414199    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83805502-d8df-466d-a7c3-c842d05347d2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414264    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414584    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414629    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414657    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.414776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.414726    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.415062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.415044    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83805502-d8df-466d-a7c3-c842d05347d2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.416771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.416753    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83805502-d8df-466d-a7c3-c842d05347d2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.416771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.416762    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83805502-d8df-466d-a7c3-c842d05347d2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.428367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.428330    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83805502-d8df-466d-a7c3-c842d05347d2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.428507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.428426    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdlvp\" (UniqueName: \"kubernetes.io/projected/83805502-d8df-466d-a7c3-c842d05347d2-kube-api-access-zdlvp\") pod \"data-science-gateway-data-science-gateway-class-68d996c5577pfxb\" (UID: \"83805502-d8df-466d-a7c3-c842d05347d2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.488611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.488516    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:03.536991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.536962    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:03.816417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:03.816327    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"]
Oct 30 19:12:03.820324 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:12:03.820298    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83805502_d8df_466d_a7c3_c842d05347d2.slice/crio-ebefa7c0e4633676735a76faa7213d6ea047e87bd11659cef3c02075b21e59c6 WatchSource:0}: Error finding container ebefa7c0e4633676735a76faa7213d6ea047e87bd11659cef3c02075b21e59c6: Status 404 returned error can't find the container with id ebefa7c0e4633676735a76faa7213d6ea047e87bd11659cef3c02075b21e59c6
Oct 30 19:12:04.710994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:04.710953    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb" event={"ID":"83805502-d8df-466d-a7c3-c842d05347d2","Type":"ContainerStarted","Data":"ebefa7c0e4633676735a76faa7213d6ea047e87bd11659cef3c02075b21e59c6"}
Oct 30 19:12:06.762002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:06.761968    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:12:06.762293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:06.762044    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:12:06.762293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:06.762073    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:12:07.726006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:07.725974    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb" event={"ID":"83805502-d8df-466d-a7c3-c842d05347d2","Type":"ContainerStarted","Data":"1b97a3e1e0487548fe27a95bcc3b99f2aabdac2c0fa1a84260d357177abaab59"}
Oct 30 19:12:07.748975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:07.748918    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb" podStartSLOduration=1.810019677 podStartE2EDuration="4.748904108s" totalImagesPullingTime="2.938884431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:07.747961177 +0000 UTC m=+362.650974213" watchObservedRunningTime="2026-10-30 19:12:07.748904108 +0000 UTC m=+362.651917145"
Oct 30 19:12:08.130402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.130322    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"]
Oct 30 19:12:08.132895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.132880    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.140237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.140210    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-cgfj7\""
Oct 30 19:12:08.141960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.141940    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 19:12:08.144038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.144017    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"]
Oct 30 19:12:08.179048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.179014    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:08.255253 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.255221    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99trd\" (UniqueName: \"kubernetes.io/projected/b4bc7918-4f7f-4441-9498-b19b27eca300-kube-api-access-99trd\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.255399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.255264    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4bc7918-4f7f-4441-9498-b19b27eca300-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.255399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.255336    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4bc7918-4f7f-4441-9498-b19b27eca300-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.255469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.255410    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4bc7918-4f7f-4441-9498-b19b27eca300-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.341223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.341192    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:08.356211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.356190    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99trd\" (UniqueName: \"kubernetes.io/projected/b4bc7918-4f7f-4441-9498-b19b27eca300-kube-api-access-99trd\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.356295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.356229    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4bc7918-4f7f-4441-9498-b19b27eca300-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.356295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.356261    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4bc7918-4f7f-4441-9498-b19b27eca300-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.356362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.356309    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4bc7918-4f7f-4441-9498-b19b27eca300-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.356671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.356655    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4bc7918-4f7f-4441-9498-b19b27eca300-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.356729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.356713    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4bc7918-4f7f-4441-9498-b19b27eca300-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.357321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.357305    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4bc7918-4f7f-4441-9498-b19b27eca300-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.368300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.368282    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99trd\" (UniqueName: \"kubernetes.io/projected/b4bc7918-4f7f-4441-9498-b19b27eca300-kube-api-access-99trd\") pod \"kserve-module-controller-manager-5cfb77bf8c-jd9rn\" (UID: \"b4bc7918-4f7f-4441-9498-b19b27eca300\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.445966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.445944    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:08.538070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.538043    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:08.542472 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.542451    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:08.723598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.723524    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"]
Oct 30 19:12:08.727360 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:12:08.727326    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bc7918_4f7f_4441_9498_b19b27eca300.slice/crio-e65c649f63c312fccdb80876dea0f8236f9194fdd50b5acf50943addbcc8843d WatchSource:0}: Error finding container e65c649f63c312fccdb80876dea0f8236f9194fdd50b5acf50943addbcc8843d: Status 404 returned error can't find the container with id e65c649f63c312fccdb80876dea0f8236f9194fdd50b5acf50943addbcc8843d
Oct 30 19:12:08.730372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.730346    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:08.731289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:08.731273    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c5577pfxb"
Oct 30 19:12:09.735704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:09.735649    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn" event={"ID":"b4bc7918-4f7f-4441-9498-b19b27eca300","Type":"ContainerStarted","Data":"e65c649f63c312fccdb80876dea0f8236f9194fdd50b5acf50943addbcc8843d"}
Oct 30 19:12:11.746654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:11.746618    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn" event={"ID":"b4bc7918-4f7f-4441-9498-b19b27eca300","Type":"ContainerStarted","Data":"915706a5235f994033e39771dc47381d4575617832e53c8f7ab1f8603d22f407"}
Oct 30 19:12:11.747044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:11.746760    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:11.767476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:11.767426    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn" podStartSLOduration=1.544089106 podStartE2EDuration="3.767410954s" totalImagesPullingTime="2.223321848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:11.766567536 +0000 UTC m=+366.669580573" watchObservedRunningTime="2026-10-30 19:12:11.767410954 +0000 UTC m=+366.670423994"
Oct 30 19:12:22.124848 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:12:22.124815    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:12:32.800199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:32.800160    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-jd9rn"
Oct 30 19:12:52.125111 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:12:52.125034    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:12:52.877306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:52.877273    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:52.968882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:52.968855    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"]
Oct 30 19:12:52.975061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:52.975041    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:12:53.000799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.000771    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jbk56"]
Oct 30 19:12:53.004058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.004039    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"]
Oct 30 19:12:53.004175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.004130    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:12:53.018059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.018041    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 19:12:53.018213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.018201    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6xg5n\""
Oct 30 19:12:53.018213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.018207    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-l7r8t\""
Oct 30 19:12:53.019813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.019797    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 19:12:53.023815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.023796    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k9pb\" (UniqueName: \"kubernetes.io/projected/a29a6635-9417-40b6-b136-bd7ed195848a-kube-api-access-4k9pb\") pod \"llmisvc-controller-manager-869d5dcc5d-ks9m8\" (UID: \"a29a6635-9417-40b6-b136-bd7ed195848a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:12:53.023927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.023913    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a29a6635-9417-40b6-b136-bd7ed195848a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ks9m8\" (UID: \"a29a6635-9417-40b6-b136-bd7ed195848a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:12:53.026951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.026931    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jbk56"]
Oct 30 19:12:53.125140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.125106    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a29a6635-9417-40b6-b136-bd7ed195848a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ks9m8\" (UID: \"a29a6635-9417-40b6-b136-bd7ed195848a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:12:53.125547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.125157    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595b9742-b806-4d07-8a18-0197d1af422e-tls-certs\") pod \"model-serving-api-75bd89fdfc-jbk56\" (UID: \"595b9742-b806-4d07-8a18-0197d1af422e\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:12:53.125547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.125183    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp7z2\" (UniqueName: \"kubernetes.io/projected/595b9742-b806-4d07-8a18-0197d1af422e-kube-api-access-sp7z2\") pod \"model-serving-api-75bd89fdfc-jbk56\" (UID: \"595b9742-b806-4d07-8a18-0197d1af422e\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:12:53.125547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.125249    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k9pb\" (UniqueName: \"kubernetes.io/projected/a29a6635-9417-40b6-b136-bd7ed195848a-kube-api-access-4k9pb\") pod \"llmisvc-controller-manager-869d5dcc5d-ks9m8\" (UID: \"a29a6635-9417-40b6-b136-bd7ed195848a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:12:53.127876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.127832    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a29a6635-9417-40b6-b136-bd7ed195848a-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ks9m8\" (UID: \"a29a6635-9417-40b6-b136-bd7ed195848a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:12:53.138859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.138836    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k9pb\" (UniqueName: \"kubernetes.io/projected/a29a6635-9417-40b6-b136-bd7ed195848a-kube-api-access-4k9pb\") pod \"llmisvc-controller-manager-869d5dcc5d-ks9m8\" (UID: \"a29a6635-9417-40b6-b136-bd7ed195848a\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:12:53.196736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.196681    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:53.226062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.226035    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595b9742-b806-4d07-8a18-0197d1af422e-tls-certs\") pod \"model-serving-api-75bd89fdfc-jbk56\" (UID: \"595b9742-b806-4d07-8a18-0197d1af422e\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:12:53.226156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.226063    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7z2\" (UniqueName: \"kubernetes.io/projected/595b9742-b806-4d07-8a18-0197d1af422e-kube-api-access-sp7z2\") pod \"model-serving-api-75bd89fdfc-jbk56\" (UID: \"595b9742-b806-4d07-8a18-0197d1af422e\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:12:53.228366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.228346    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595b9742-b806-4d07-8a18-0197d1af422e-tls-certs\") pod \"model-serving-api-75bd89fdfc-jbk56\" (UID: \"595b9742-b806-4d07-8a18-0197d1af422e\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:12:53.237989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.237968    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7z2\" (UniqueName: \"kubernetes.io/projected/595b9742-b806-4d07-8a18-0197d1af422e-kube-api-access-sp7z2\") pod \"model-serving-api-75bd89fdfc-jbk56\" (UID: \"595b9742-b806-4d07-8a18-0197d1af422e\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:12:53.287437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.287420    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:12:53.316781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.316758    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:12:53.386026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.385199    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:53.570982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.570956    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"]
Oct 30 19:12:53.574120 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:12:53.574097    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda29a6635_9417_40b6_b136_bd7ed195848a.slice/crio-3c2220f5780980528aaf965e7355599a73e856e4dd251d89416661833d87038a WatchSource:0}: Error finding container 3c2220f5780980528aaf965e7355599a73e856e4dd251d89416661833d87038a: Status 404 returned error can't find the container with id 3c2220f5780980528aaf965e7355599a73e856e4dd251d89416661833d87038a
Oct 30 19:12:53.599509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.599474    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jbk56"]
Oct 30 19:12:53.603069 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:12:53.603034    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595b9742_b806_4d07_8a18_0197d1af422e.slice/crio-973e53e79a4a2040788ee31a58320d14f20a2689bfb36ea992fbc78f75c87e33 WatchSource:0}: Error finding container 973e53e79a4a2040788ee31a58320d14f20a2689bfb36ea992fbc78f75c87e33: Status 404 returned error can't find the container with id 973e53e79a4a2040788ee31a58320d14f20a2689bfb36ea992fbc78f75c87e33
Oct 30 19:12:53.919444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.919404    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jbk56" event={"ID":"595b9742-b806-4d07-8a18-0197d1af422e","Type":"ContainerStarted","Data":"973e53e79a4a2040788ee31a58320d14f20a2689bfb36ea992fbc78f75c87e33"}
Oct 30 19:12:53.920573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:53.920546    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8" event={"ID":"a29a6635-9417-40b6-b136-bd7ed195848a","Type":"ContainerStarted","Data":"3c2220f5780980528aaf965e7355599a73e856e4dd251d89416661833d87038a"}
Oct 30 19:12:54.918478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:54.918438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:55.100083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:55.100042    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:55.269434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:55.269322    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:55.931532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:55.931501    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jbk56" event={"ID":"595b9742-b806-4d07-8a18-0197d1af422e","Type":"ContainerStarted","Data":"0c65d415c90443a90748108300479eca50f635c6a0adfd8af7fa246823303c7c"}
Oct 30 19:12:55.932031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:55.931582    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:12:55.933079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:55.933055    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8" event={"ID":"a29a6635-9417-40b6-b136-bd7ed195848a","Type":"ContainerStarted","Data":"b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff"}
Oct 30 19:12:55.933198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:55.933154    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:12:55.950622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:55.950579    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-jbk56" podStartSLOduration=2.148394034 podStartE2EDuration="3.950567583s" totalImagesPullingTime="1.802173549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:55.949070145 +0000 UTC m=+410.852083183" watchObservedRunningTime="2026-10-30 19:12:55.950567583 +0000 UTC m=+410.853580619"
Oct 30 19:12:55.965457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:12:55.965421    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8" podStartSLOduration=1.9866869 podStartE2EDuration="3.965410347s" totalImagesPullingTime="1.978723447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:55.965120751 +0000 UTC m=+410.868133789" watchObservedRunningTime="2026-10-30 19:12:55.965410347 +0000 UTC m=+410.868423380"
Oct 30 19:13:06.942263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:13:06.942237    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-jbk56"
Oct 30 19:13:22.124245 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:13:22.124211    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:13:26.941059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:13:26.941027    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:13:29.906676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:13:29.906643    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:30.074227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:13:30.074192    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:49.522287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:13:49.522248    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:49.714755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:13:49.714719    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:52.124823 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:13:52.124789    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:13:55.273766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:13:55.273730    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:07.535889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:07.535857    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:08.353782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:08.353749    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:09.142027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:09.141983    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:12.283337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:12.283304    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:12.484968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:12.484937    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:12.900754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:12.900723    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:13.697224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:13.697190    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:13.907714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:13.907660    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:14.752114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:14.752086    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"]
Oct 30 19:14:14.755348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:14.755332    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"
Oct 30 19:14:14.766960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:14.766939    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n4z59\""
Oct 30 19:14:14.767208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:14.767184    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:14:14.767311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:14.767189    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:14:14.768606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:14.768586    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"]
Oct 30 19:14:14.901723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:14.901672    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5vk\" (UniqueName: \"kubernetes.io/projected/27be6c75-203d-49a0-a643-0b2fa5951724-kube-api-access-ks5vk\") pod \"authorino-operator-68d6dd76dc-qxmlg\" (UID: \"27be6c75-203d-49a0-a643-0b2fa5951724\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"
Oct 30 19:14:15.002063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:15.002031    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5vk\" (UniqueName: \"kubernetes.io/projected/27be6c75-203d-49a0-a643-0b2fa5951724-kube-api-access-ks5vk\") pod \"authorino-operator-68d6dd76dc-qxmlg\" (UID: \"27be6c75-203d-49a0-a643-0b2fa5951724\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"
Oct 30 19:14:15.014530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:15.014480    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5vk\" (UniqueName: \"kubernetes.io/projected/27be6c75-203d-49a0-a643-0b2fa5951724-kube-api-access-ks5vk\") pod \"authorino-operator-68d6dd76dc-qxmlg\" (UID: \"27be6c75-203d-49a0-a643-0b2fa5951724\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"
Oct 30 19:14:15.067210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:15.067178    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"
Oct 30 19:14:15.343442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:15.343413    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"]
Oct 30 19:14:15.345969 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:14:15.345945    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27be6c75_203d_49a0_a643_0b2fa5951724.slice/crio-614a4f005260b0482713453599aba0111851d31a84f4a1ee74b588f6ebf7da4c WatchSource:0}: Error finding container 614a4f005260b0482713453599aba0111851d31a84f4a1ee74b588f6ebf7da4c: Status 404 returned error can't find the container with id 614a4f005260b0482713453599aba0111851d31a84f4a1ee74b588f6ebf7da4c
Oct 30 19:14:15.590826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:15.590791    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:15.778674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:15.778640    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:16.261457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:16.261422    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg" event={"ID":"27be6c75-203d-49a0-a643-0b2fa5951724","Type":"ContainerStarted","Data":"614a4f005260b0482713453599aba0111851d31a84f4a1ee74b588f6ebf7da4c"}
Oct 30 19:14:17.797064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:17.797020    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:18.391847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:18.391817    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:19.275354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:19.275318    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg" event={"ID":"27be6c75-203d-49a0-a643-0b2fa5951724","Type":"ContainerStarted","Data":"46688ff8c415e1bbdf4b9a545160d5ef90c7e1b1a91c6800e43e932aec4c0386"}
Oct 30 19:14:19.275793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:19.275371    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"
Oct 30 19:14:19.298300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:19.298228    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg" podStartSLOduration=2.38828781 podStartE2EDuration="5.298211851s" totalImagesPullingTime="2.909924041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:19.297187682 +0000 UTC m=+494.200200719" watchObservedRunningTime="2026-10-30 19:14:19.298211851 +0000 UTC m=+494.201224889"
Oct 30 19:14:22.124455 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:14:22.124420    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:14:22.758092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:22.758054    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:22.930220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:22.930192    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:23.145592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:23.145560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:23.304159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:23.304124    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:23.548882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:23.548789    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:23.773502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:23.773472    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:24.142939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.142913    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"]
Oct 30 19:14:24.146375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.146355    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.156076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.156048    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"]
Oct 30 19:14:24.159116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.159097    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 19:14:24.160428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.160412    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zs2nd\""
Oct 30 19:14:24.160428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.160414    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 19:14:24.227801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.227770    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:24.274580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.274549    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a43fae4-0624-46f6-9ea8-39770dbf7afb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-hzxfv\" (UID: \"9a43fae4-0624-46f6-9ea8-39770dbf7afb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.274783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.274617    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kczgm\" (UniqueName: \"kubernetes.io/projected/9a43fae4-0624-46f6-9ea8-39770dbf7afb-kube-api-access-kczgm\") pod \"kuadrant-console-plugin-66c4fb575d-hzxfv\" (UID: \"9a43fae4-0624-46f6-9ea8-39770dbf7afb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.274783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.274668    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a43fae4-0624-46f6-9ea8-39770dbf7afb-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-hzxfv\" (UID: \"9a43fae4-0624-46f6-9ea8-39770dbf7afb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.375496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.375461    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kczgm\" (UniqueName: \"kubernetes.io/projected/9a43fae4-0624-46f6-9ea8-39770dbf7afb-kube-api-access-kczgm\") pod \"kuadrant-console-plugin-66c4fb575d-hzxfv\" (UID: \"9a43fae4-0624-46f6-9ea8-39770dbf7afb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.375496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.375493    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a43fae4-0624-46f6-9ea8-39770dbf7afb-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-hzxfv\" (UID: \"9a43fae4-0624-46f6-9ea8-39770dbf7afb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.375790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.375597    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a43fae4-0624-46f6-9ea8-39770dbf7afb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-hzxfv\" (UID: \"9a43fae4-0624-46f6-9ea8-39770dbf7afb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.376320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.376297    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a43fae4-0624-46f6-9ea8-39770dbf7afb-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-hzxfv\" (UID: \"9a43fae4-0624-46f6-9ea8-39770dbf7afb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.378250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.378228    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a43fae4-0624-46f6-9ea8-39770dbf7afb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-hzxfv\" (UID: \"9a43fae4-0624-46f6-9ea8-39770dbf7afb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.387505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.387479    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kczgm\" (UniqueName: \"kubernetes.io/projected/9a43fae4-0624-46f6-9ea8-39770dbf7afb-kube-api-access-kczgm\") pod \"kuadrant-console-plugin-66c4fb575d-hzxfv\" (UID: \"9a43fae4-0624-46f6-9ea8-39770dbf7afb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.388173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.388146    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:24.459192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.459117    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"
Oct 30 19:14:24.731917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:24.731843    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv"]
Oct 30 19:14:24.735366 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:14:24.735340    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a43fae4_0624_46f6_9ea8_39770dbf7afb.slice/crio-a4242e3c5b87cc7e7ef9f6de845dac8b8b010133435d72f15387137c431a762b WatchSource:0}: Error finding container a4242e3c5b87cc7e7ef9f6de845dac8b8b010133435d72f15387137c431a762b: Status 404 returned error can't find the container with id a4242e3c5b87cc7e7ef9f6de845dac8b8b010133435d72f15387137c431a762b
Oct 30 19:14:25.305294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:25.305263    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv" event={"ID":"9a43fae4-0624-46f6-9ea8-39770dbf7afb","Type":"ContainerStarted","Data":"a4242e3c5b87cc7e7ef9f6de845dac8b8b010133435d72f15387137c431a762b"}
Oct 30 19:14:30.282089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:30.282061    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-qxmlg"
Oct 30 19:14:31.332954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:31.332921    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv" event={"ID":"9a43fae4-0624-46f6-9ea8-39770dbf7afb","Type":"ContainerStarted","Data":"a52f0a456ef5efc0b6c17f74c3d39073d490113120b1f4f92df725f58a7d804c"}
Oct 30 19:14:31.350043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:31.349990    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-hzxfv" podStartSLOduration=1.734037196 podStartE2EDuration="7.349976925s" totalImagesPullingTime="5.615939729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:31.34916081 +0000 UTC m=+506.252173848" watchObservedRunningTime="2026-10-30 19:14:31.349976925 +0000 UTC m=+506.252989962"
Oct 30 19:14:31.571909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:31.571875    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:31.743682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:31.743625    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:34.718325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:34.718293    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:34.916522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:34.916488    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:37.103355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:37.103318    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:37.301018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:37.300990    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:43.947676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:43.947636    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:44.130418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:44.130381    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:44.376566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:44.376481    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:44.545284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:44.545244    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:44.722984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:44.722950    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:52.124205 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:14:52.124171    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:14:52.984090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:52.984050    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:53.154816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:53.154781    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:55.337860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:55.337810    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:55.961807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:55.961770    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:56.201433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:56.201402    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:58.420834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:58.420800    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:58.607595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:14:58.607557    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:06.714255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.714220    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"]
Oct 30 19:15:06.737533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.737495    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"]
Oct 30 19:15:06.737742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.737592    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.784807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.784778    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vdgcw\""
Oct 30 19:15:06.813908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.813872    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.814072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.813920    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.814072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.813944    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.814072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.814018    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.814072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.814049    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.814204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.814075    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.814204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.814160    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.814204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.814179    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwx9d\" (UniqueName: \"kubernetes.io/projected/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-kube-api-access-wwx9d\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.814296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.814250    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.816528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.816504    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:06.915118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915081    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915112    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwx9d\" (UniqueName: \"kubernetes.io/projected/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-kube-api-access-wwx9d\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915135    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915183    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915221    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915246    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915294    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915330    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915364    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915545    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.915961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915938    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.916238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915978    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.916238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.915994    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.916238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.916170    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.917901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.917870    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.918133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.918116    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.927022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.927002    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.927423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.927401    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwx9d\" (UniqueName: \"kubernetes.io/projected/9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f-kube-api-access-wwx9d\") pod \"maas-default-gateway-openshift-default-855f9574cf-cllrc\" (UID: \"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:06.978158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:06.978068    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:07.050888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:07.050852    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:07.336895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:07.336808    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"]
Oct 30 19:15:07.339682 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:15:07.339653    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1a9e9f_4030_4fc5_b7cc_a1be1aab6b4f.slice/crio-cf900626cc8bc61cff3432ab84971907d1125a4fc51f16ea8a8691e44baad838 WatchSource:0}: Error finding container cf900626cc8bc61cff3432ab84971907d1125a4fc51f16ea8a8691e44baad838: Status 404 returned error can't find the container with id cf900626cc8bc61cff3432ab84971907d1125a4fc51f16ea8a8691e44baad838
Oct 30 19:15:07.341819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:07.341786    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:15:07.341899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:07.341854    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:15:07.341899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:07.341888    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:15:07.484965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:07.484924    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" event={"ID":"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f","Type":"ContainerStarted","Data":"e1b157d84f1085157e45c3111dd08b4e1afdce45bd9d72d3742c5437eb157f17"}
Oct 30 19:15:07.485113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:07.484993    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" event={"ID":"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f","Type":"ContainerStarted","Data":"cf900626cc8bc61cff3432ab84971907d1125a4fc51f16ea8a8691e44baad838"}
Oct 30 19:15:07.507026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:07.506971    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" podStartSLOduration=1.506957865 podStartE2EDuration="1.506957865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:07.506137944 +0000 UTC m=+542.409150981" watchObservedRunningTime="2026-10-30 19:15:07.506957865 +0000 UTC m=+542.409970901"
Oct 30 19:15:08.051134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:08.051104    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:08.055751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:08.055730    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:08.489357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:08.489325    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:08.490214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:08.490192    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:15:17.889629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:17.889592    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:18.059418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:18.059384    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:18.703742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:18.703708    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:18.866042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:18.866008    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:19.440738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.440708    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kw822"]
Oct 30 19:15:19.443962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.443942    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kw822"
Oct 30 19:15:19.449936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.449893    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5tbl8\""
Oct 30 19:15:19.450240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.450218    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kw822"]
Oct 30 19:15:19.520242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.520212    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpl5\" (UniqueName: \"kubernetes.io/projected/7c24b146-18f6-40e3-9e6d-c179cd7f7d18-kube-api-access-6xpl5\") pod \"authorino-57f7f8659f-kw822\" (UID: \"7c24b146-18f6-40e3-9e6d-c179cd7f7d18\") " pod="kuadrant-system/authorino-57f7f8659f-kw822"
Oct 30 19:15:19.563982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.563948    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:19.621102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.621077    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpl5\" (UniqueName: \"kubernetes.io/projected/7c24b146-18f6-40e3-9e6d-c179cd7f7d18-kube-api-access-6xpl5\") pod \"authorino-57f7f8659f-kw822\" (UID: \"7c24b146-18f6-40e3-9e6d-c179cd7f7d18\") " pod="kuadrant-system/authorino-57f7f8659f-kw822"
Oct 30 19:15:19.636443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.636415    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpl5\" (UniqueName: \"kubernetes.io/projected/7c24b146-18f6-40e3-9e6d-c179cd7f7d18-kube-api-access-6xpl5\") pod \"authorino-57f7f8659f-kw822\" (UID: \"7c24b146-18f6-40e3-9e6d-c179cd7f7d18\") " pod="kuadrant-system/authorino-57f7f8659f-kw822"
Oct 30 19:15:19.734769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.734675    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:19.756580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:19.756558    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kw822"
Oct 30 19:15:20.026630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:20.026560    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kw822"]
Oct 30 19:15:20.030413 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:15:20.030385    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c24b146_18f6_40e3_9e6d_c179cd7f7d18.slice/crio-6db78a7b5f1d4496356ede1684091fdd11fbd93b5bcc1cd8811fff1958df2d9a WatchSource:0}: Error finding container 6db78a7b5f1d4496356ede1684091fdd11fbd93b5bcc1cd8811fff1958df2d9a: Status 404 returned error can't find the container with id 6db78a7b5f1d4496356ede1684091fdd11fbd93b5bcc1cd8811fff1958df2d9a
Oct 30 19:15:20.540939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:20.540901    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kw822" event={"ID":"7c24b146-18f6-40e3-9e6d-c179cd7f7d18","Type":"ContainerStarted","Data":"6db78a7b5f1d4496356ede1684091fdd11fbd93b5bcc1cd8811fff1958df2d9a"}
Oct 30 19:15:22.128401 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:15:22.128365    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:15:23.317065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:23.316980    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kw822"]
Oct 30 19:15:23.554865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:23.554834    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kw822" event={"ID":"7c24b146-18f6-40e3-9e6d-c179cd7f7d18","Type":"ContainerStarted","Data":"37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e"}
Oct 30 19:15:23.569438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:23.569348    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-kw822" podStartSLOduration=1.663700298 podStartE2EDuration="4.569329673s" totalImagesPullingTime="2.905629375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:23.568921825 +0000 UTC m=+558.471934864" watchObservedRunningTime="2026-10-30 19:15:23.569329673 +0000 UTC m=+558.472342711"
Oct 30 19:15:24.559069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:24.559036    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-kw822" podUID="7c24b146-18f6-40e3-9e6d-c179cd7f7d18" containerName="authorino" containerID="cri-o://37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e" gracePeriod=30
Oct 30 19:15:24.812159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:24.812102    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kw822"
Oct 30 19:15:24.968429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:24.968391    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c24b146-18f6-40e3-9e6d-c179cd7f7d18-kube-api-access-6xpl5\" (UniqueName: \"kubernetes.io/projected/7c24b146-18f6-40e3-9e6d-c179cd7f7d18-kube-api-access-6xpl5\") pod \"7c24b146-18f6-40e3-9e6d-c179cd7f7d18\" (UID: \"7c24b146-18f6-40e3-9e6d-c179cd7f7d18\") "
Oct 30 19:15:24.970767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:24.970740    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c24b146-18f6-40e3-9e6d-c179cd7f7d18-kube-api-access-6xpl5" pod "7c24b146-18f6-40e3-9e6d-c179cd7f7d18" (UID: "7c24b146-18f6-40e3-9e6d-c179cd7f7d18"). InnerVolumeSpecName "kube-api-access-6xpl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:25.069868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.069791    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xpl5\" (UniqueName: \"kubernetes.io/projected/7c24b146-18f6-40e3-9e6d-c179cd7f7d18-kube-api-access-6xpl5\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:25.564452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.564421    4925 generic.go:356] "Generic (PLEG): container finished" podID="7c24b146-18f6-40e3-9e6d-c179cd7f7d18" containerID="37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e" exitCode=0
Oct 30 19:15:25.564877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.564484    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kw822" event={"ID":"7c24b146-18f6-40e3-9e6d-c179cd7f7d18","Type":"ContainerDied","Data":"37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e"}
Oct 30 19:15:25.564877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.564516    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kw822" event={"ID":"7c24b146-18f6-40e3-9e6d-c179cd7f7d18","Type":"ContainerDied","Data":"6db78a7b5f1d4496356ede1684091fdd11fbd93b5bcc1cd8811fff1958df2d9a"}
Oct 30 19:15:25.564877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.564523    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kw822"
Oct 30 19:15:25.564877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.564531    4925 scope.go:122] "RemoveContainer" containerID="37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e"
Oct 30 19:15:25.572883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.572867    4925 scope.go:122] "RemoveContainer" containerID="37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e"
Oct 30 19:15:25.573142 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:15:25.573124    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e\": container with ID starting with 37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e not found: ID does not exist" containerID="37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e"
Oct 30 19:15:25.573201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.573156    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e"} err="failed to get container status \"37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e\": rpc error: code = NotFound desc = could not find container \"37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e\": container with ID starting with 37a5525e41ccd26dc4ea32c53ac113a664ff5d3e196118412b6e513eaad1285e not found: ID does not exist"
Oct 30 19:15:25.578556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.578536    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kw822"]
Oct 30 19:15:25.582937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:25.582921    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kw822"]
Oct 30 19:15:27.261045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:27.261015    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c24b146-18f6-40e3-9e6d-c179cd7f7d18" path="/var/lib/kubelet/pods/7c24b146-18f6-40e3-9e6d-c179cd7f7d18/volumes"
Oct 30 19:15:35.988445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:35.988396    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:37.679557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:37.679514    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:41.609641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:41.609608    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:41.836224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:41.836188    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:51.384577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.384485    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:51.550976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.550944    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:51.629293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.629265    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-9g58n"]
Oct 30 19:15:51.629629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.629616    4925 cpu_manager.go:414] "containerMap: removing container" podUID="7c24b146-18f6-40e3-9e6d-c179cd7f7d18" containerName="authorino"
Oct 30 19:15:51.629679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.629630    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c24b146-18f6-40e3-9e6d-c179cd7f7d18" containerName="authorino"
Oct 30 19:15:51.629764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.629744    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c24b146-18f6-40e3-9e6d-c179cd7f7d18" containerName="authorino"
Oct 30 19:15:51.632349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.632335    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:51.639512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.639488    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5tbl8\""
Oct 30 19:15:51.639743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.639721    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 19:15:51.641331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.641312    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-9g58n"]
Oct 30 19:15:51.667086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.667062    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1690e47-6ef9-413e-881d-0f4c982d996d-tls-cert\") pod \"authorino-6dd76875b5-9g58n\" (UID: \"a1690e47-6ef9-413e-881d-0f4c982d996d\") " pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:51.667205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.667116    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzhbd\" (UniqueName: \"kubernetes.io/projected/a1690e47-6ef9-413e-881d-0f4c982d996d-kube-api-access-fzhbd\") pod \"authorino-6dd76875b5-9g58n\" (UID: \"a1690e47-6ef9-413e-881d-0f4c982d996d\") " pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:51.670606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.670572    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-9g58n"]
Oct 30 19:15:51.671164 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:15:51.671135    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fzhbd tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-9g58n" podUID="a1690e47-6ef9-413e-881d-0f4c982d996d"
Oct 30 19:15:51.702050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.702024    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-4b9f8"]
Oct 30 19:15:51.705168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.705153    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:51.713194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.713171    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-4b9f8"]
Oct 30 19:15:51.768014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.767984    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzhbd\" (UniqueName: \"kubernetes.io/projected/a1690e47-6ef9-413e-881d-0f4c982d996d-kube-api-access-fzhbd\") pod \"authorino-6dd76875b5-9g58n\" (UID: \"a1690e47-6ef9-413e-881d-0f4c982d996d\") " pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:51.768178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.768024    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2xs\" (UniqueName: \"kubernetes.io/projected/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-kube-api-access-kz2xs\") pod \"authorino-6889cbd8c4-4b9f8\" (UID: \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\") " pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:51.768178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.768096    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1690e47-6ef9-413e-881d-0f4c982d996d-tls-cert\") pod \"authorino-6dd76875b5-9g58n\" (UID: \"a1690e47-6ef9-413e-881d-0f4c982d996d\") " pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:51.768178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.768124    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-tls-cert\") pod \"authorino-6889cbd8c4-4b9f8\" (UID: \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\") " pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:51.770649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.770624    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1690e47-6ef9-413e-881d-0f4c982d996d-tls-cert\") pod \"authorino-6dd76875b5-9g58n\" (UID: \"a1690e47-6ef9-413e-881d-0f4c982d996d\") " pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:51.779851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.779827    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzhbd\" (UniqueName: \"kubernetes.io/projected/a1690e47-6ef9-413e-881d-0f4c982d996d-kube-api-access-fzhbd\") pod \"authorino-6dd76875b5-9g58n\" (UID: \"a1690e47-6ef9-413e-881d-0f4c982d996d\") " pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:51.869574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.869544    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-tls-cert\") pod \"authorino-6889cbd8c4-4b9f8\" (UID: \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\") " pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:51.869742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.869606    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2xs\" (UniqueName: \"kubernetes.io/projected/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-kube-api-access-kz2xs\") pod \"authorino-6889cbd8c4-4b9f8\" (UID: \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\") " pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:51.872228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.872205    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-tls-cert\") pod \"authorino-6889cbd8c4-4b9f8\" (UID: \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\") " pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:51.881425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.881404    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2xs\" (UniqueName: \"kubernetes.io/projected/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-kube-api-access-kz2xs\") pod \"authorino-6889cbd8c4-4b9f8\" (UID: \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\") " pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:51.988521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.988495    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-4b9f8"]
Oct 30 19:15:51.988698 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:51.988674    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:52.020772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.020750    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6fdffc46cf-sq9nd"]
Oct 30 19:15:52.024337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.024320    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:15:52.032929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.032904    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fdffc46cf-sq9nd"]
Oct 30 19:15:52.071709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.071658    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2njcr\" (UniqueName: \"kubernetes.io/projected/cb66ba6d-857a-4a36-a920-c874426bba28-kube-api-access-2njcr\") pod \"authorino-6fdffc46cf-sq9nd\" (UID: \"cb66ba6d-857a-4a36-a920-c874426bba28\") " pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:15:52.071851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.071800    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb66ba6d-857a-4a36-a920-c874426bba28-tls-cert\") pod \"authorino-6fdffc46cf-sq9nd\" (UID: \"cb66ba6d-857a-4a36-a920-c874426bba28\") " pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:15:52.124634 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:15:52.124601    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:15:52.173201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.173177    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njcr\" (UniqueName: \"kubernetes.io/projected/cb66ba6d-857a-4a36-a920-c874426bba28-kube-api-access-2njcr\") pod \"authorino-6fdffc46cf-sq9nd\" (UID: \"cb66ba6d-857a-4a36-a920-c874426bba28\") " pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:15:52.173337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.173242    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb66ba6d-857a-4a36-a920-c874426bba28-tls-cert\") pod \"authorino-6fdffc46cf-sq9nd\" (UID: \"cb66ba6d-857a-4a36-a920-c874426bba28\") " pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:15:52.175587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.175568    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb66ba6d-857a-4a36-a920-c874426bba28-tls-cert\") pod \"authorino-6fdffc46cf-sq9nd\" (UID: \"cb66ba6d-857a-4a36-a920-c874426bba28\") " pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:15:52.184464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.184435    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njcr\" (UniqueName: \"kubernetes.io/projected/cb66ba6d-857a-4a36-a920-c874426bba28-kube-api-access-2njcr\") pod \"authorino-6fdffc46cf-sq9nd\" (UID: \"cb66ba6d-857a-4a36-a920-c874426bba28\") " pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:15:52.262675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.262611    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-4b9f8"]
Oct 30 19:15:52.266083 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:15:52.266056    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bad8ef4_9714_4fa4_88d7_210d7740e9d5.slice/crio-f9c4aa35b04ff1ef02572011fb6fc995e91185bc92adc553771ad61cc9eadb1d WatchSource:0}: Error finding container f9c4aa35b04ff1ef02572011fb6fc995e91185bc92adc553771ad61cc9eadb1d: Status 404 returned error can't find the container with id f9c4aa35b04ff1ef02572011fb6fc995e91185bc92adc553771ad61cc9eadb1d
Oct 30 19:15:52.339638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.339610    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:15:52.612816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.612733    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fdffc46cf-sq9nd"]
Oct 30 19:15:52.616049 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:15:52.616021    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb66ba6d_857a_4a36_a920_c874426bba28.slice/crio-c58acb5f3e265d803d88f37f54490c206f52730102bfc633c574d07f5a7b5b2a WatchSource:0}: Error finding container c58acb5f3e265d803d88f37f54490c206f52730102bfc633c574d07f5a7b5b2a: Status 404 returned error can't find the container with id c58acb5f3e265d803d88f37f54490c206f52730102bfc633c574d07f5a7b5b2a
Oct 30 19:15:52.683943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.683910    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8" event={"ID":"7bad8ef4-9714-4fa4-88d7-210d7740e9d5","Type":"ContainerStarted","Data":"f9c4aa35b04ff1ef02572011fb6fc995e91185bc92adc553771ad61cc9eadb1d"}
Oct 30 19:15:52.684979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.684953    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd" event={"ID":"cb66ba6d-857a-4a36-a920-c874426bba28","Type":"ContainerStarted","Data":"c58acb5f3e265d803d88f37f54490c206f52730102bfc633c574d07f5a7b5b2a"}
Oct 30 19:15:52.685100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.684986    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:52.692427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.692401    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:52.778086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.778064    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1690e47-6ef9-413e-881d-0f4c982d996d-kube-api-access-fzhbd\" (UniqueName: \"kubernetes.io/projected/a1690e47-6ef9-413e-881d-0f4c982d996d-kube-api-access-fzhbd\") pod \"a1690e47-6ef9-413e-881d-0f4c982d996d\" (UID: \"a1690e47-6ef9-413e-881d-0f4c982d996d\") "
Oct 30 19:15:52.778196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.778158    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1690e47-6ef9-413e-881d-0f4c982d996d-tls-cert\" (UniqueName: \"kubernetes.io/secret/a1690e47-6ef9-413e-881d-0f4c982d996d-tls-cert\") pod \"a1690e47-6ef9-413e-881d-0f4c982d996d\" (UID: \"a1690e47-6ef9-413e-881d-0f4c982d996d\") "
Oct 30 19:15:52.780299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.780275    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1690e47-6ef9-413e-881d-0f4c982d996d-tls-cert" pod "a1690e47-6ef9-413e-881d-0f4c982d996d" (UID: "a1690e47-6ef9-413e-881d-0f4c982d996d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:52.780392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.780347    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1690e47-6ef9-413e-881d-0f4c982d996d-kube-api-access-fzhbd" pod "a1690e47-6ef9-413e-881d-0f4c982d996d" (UID: "a1690e47-6ef9-413e-881d-0f4c982d996d"). InnerVolumeSpecName "kube-api-access-fzhbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:52.879391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.879333    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzhbd\" (UniqueName: \"kubernetes.io/projected/a1690e47-6ef9-413e-881d-0f4c982d996d-kube-api-access-fzhbd\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:52.879391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:52.879355    4925 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1690e47-6ef9-413e-881d-0f4c982d996d-tls-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:53.690560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:53.690520    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd" event={"ID":"cb66ba6d-857a-4a36-a920-c874426bba28","Type":"ContainerStarted","Data":"6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2"}
Oct 30 19:15:53.691060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:53.690667    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-9g58n"
Oct 30 19:15:53.707110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:53.707065    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd" podStartSLOduration=1.969369352 podStartE2EDuration="2.707052636s" totalImagesPullingTime="737.683284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:53.705936538 +0000 UTC m=+588.608949574" watchObservedRunningTime="2026-10-30 19:15:53.707052636 +0000 UTC m=+588.610065672"
Oct 30 19:15:53.728939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:53.728914    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-9g58n"]
Oct 30 19:15:53.744135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:53.744106    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-9g58n"]
Oct 30 19:15:54.697277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:54.697227    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8" event={"ID":"7bad8ef4-9714-4fa4-88d7-210d7740e9d5","Type":"ContainerStarted","Data":"72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d"}
Oct 30 19:15:54.697760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:54.697286    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8" podUID="7bad8ef4-9714-4fa4-88d7-210d7740e9d5" containerName="authorino" containerID="cri-o://72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d" gracePeriod=30
Oct 30 19:15:54.722241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:54.722197    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8" podStartSLOduration=2.197430721 podStartE2EDuration="3.722182109s" totalImagesPullingTime="1.524751388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:54.719282171 +0000 UTC m=+589.622295208" watchObservedRunningTime="2026-10-30 19:15:54.722182109 +0000 UTC m=+589.625195183"
Oct 30 19:15:54.953590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:54.953531    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:54.996804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:54.996768    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-kube-api-access-kz2xs\" (UniqueName: \"kubernetes.io/projected/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-kube-api-access-kz2xs\") pod \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\" (UID: \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\") "
Oct 30 19:15:54.996994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:54.996816    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-tls-cert\" (UniqueName: \"kubernetes.io/secret/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-tls-cert\") pod \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\" (UID: \"7bad8ef4-9714-4fa4-88d7-210d7740e9d5\") "
Oct 30 19:15:54.999223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:54.999191    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-kube-api-access-kz2xs" pod "7bad8ef4-9714-4fa4-88d7-210d7740e9d5" (UID: "7bad8ef4-9714-4fa4-88d7-210d7740e9d5"). InnerVolumeSpecName "kube-api-access-kz2xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:55.007726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.007700    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-tls-cert" pod "7bad8ef4-9714-4fa4-88d7-210d7740e9d5" (UID: "7bad8ef4-9714-4fa4-88d7-210d7740e9d5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:55.098016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.097987    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2xs\" (UniqueName: \"kubernetes.io/projected/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-kube-api-access-kz2xs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:55.098016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.098014    4925 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7bad8ef4-9714-4fa4-88d7-210d7740e9d5-tls-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:55.261619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.261551    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1690e47-6ef9-413e-881d-0f4c982d996d" path="/var/lib/kubelet/pods/a1690e47-6ef9-413e-881d-0f4c982d996d/volumes"
Oct 30 19:15:55.398811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.398772    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:55.703274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.703237    4925 generic.go:356] "Generic (PLEG): container finished" podID="7bad8ef4-9714-4fa4-88d7-210d7740e9d5" containerID="72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d" exitCode=0
Oct 30 19:15:55.703759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.703282    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8"
Oct 30 19:15:55.703759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.703316    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8" event={"ID":"7bad8ef4-9714-4fa4-88d7-210d7740e9d5","Type":"ContainerDied","Data":"72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d"}
Oct 30 19:15:55.703759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.703356    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-4b9f8" event={"ID":"7bad8ef4-9714-4fa4-88d7-210d7740e9d5","Type":"ContainerDied","Data":"f9c4aa35b04ff1ef02572011fb6fc995e91185bc92adc553771ad61cc9eadb1d"}
Oct 30 19:15:55.703759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.703379    4925 scope.go:122] "RemoveContainer" containerID="72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d"
Oct 30 19:15:55.714373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.714353    4925 scope.go:122] "RemoveContainer" containerID="72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d"
Oct 30 19:15:55.714670 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:15:55.714651    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d\": container with ID starting with 72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d not found: ID does not exist" containerID="72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d"
Oct 30 19:15:55.714745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.714680    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d"} err="failed to get container status \"72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d\": rpc error: code = NotFound desc = could not find container \"72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d\": container with ID starting with 72ee8b893e215c207c5993ccb1d2d291b54e56cbb0e64c3450b6041d1467ae4d not found: ID does not exist"
Oct 30 19:15:55.717557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.717537    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-4b9f8"]
Oct 30 19:15:55.721235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:55.721217    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-4b9f8"]
Oct 30 19:15:57.260994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:15:57.260964    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bad8ef4-9714-4fa4-88d7-210d7740e9d5" path="/var/lib/kubelet/pods/7bad8ef4-9714-4fa4-88d7-210d7740e9d5/volumes"
Oct 30 19:16:01.213509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:01.213468    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:01.386732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:01.386671    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:01.554355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:01.554268    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:05.260986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:05.260951    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:05.298009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:05.297975    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:16:05.298220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:05.298200    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:16:05.427721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:05.427672    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:08.310367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:08.310324    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:08.473865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:08.473829    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:22.124511 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:16:22.124474    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:16:35.071323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:35.071289    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:35.297273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:35.297236    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:36.249704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:36.249655    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:36.420929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:36.420897    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:36.595470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:36.595394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:46.947220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:46.947136    4925 generic.go:356] "Generic (PLEG): container finished" podID="9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f" containerID="e1b157d84f1085157e45c3111dd08b4e1afdce45bd9d72d3742c5437eb157f17" exitCode=0
Oct 30 19:16:46.947220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:46.947209    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" event={"ID":"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f","Type":"ContainerDied","Data":"e1b157d84f1085157e45c3111dd08b4e1afdce45bd9d72d3742c5437eb157f17"}
Oct 30 19:16:46.947820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:46.947668    4925 scope.go:122] "RemoveContainer" containerID="e1b157d84f1085157e45c3111dd08b4e1afdce45bd9d72d3742c5437eb157f17"
Oct 30 19:16:46.948790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:46.948759    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:16:46.948890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:46.948827    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:16:46.948890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:46.948867    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:16:47.240503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.240475    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"]
Oct 30 19:16:47.240942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.240919    4925 cpu_manager.go:414] "containerMap: removing container" podUID="7bad8ef4-9714-4fa4-88d7-210d7740e9d5" containerName="authorino"
Oct 30 19:16:47.240942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.240941    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bad8ef4-9714-4fa4-88d7-210d7740e9d5" containerName="authorino"
Oct 30 19:16:47.241158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.241082    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bad8ef4-9714-4fa4-88d7-210d7740e9d5" containerName="authorino"
Oct 30 19:16:47.243046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.243028    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:16:47.254044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.254020    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"]
Oct 30 19:16:47.260124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.260088    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26772c06-318e-40d8-a330-fc84c521a37d-cert\") pod \"llmisvc-controller-manager-6466c554d8-vzg9h\" (UID: \"26772c06-318e-40d8-a330-fc84c521a37d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:16:47.260242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.260147    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znj98\" (UniqueName: \"kubernetes.io/projected/26772c06-318e-40d8-a330-fc84c521a37d-kube-api-access-znj98\") pod \"llmisvc-controller-manager-6466c554d8-vzg9h\" (UID: \"26772c06-318e-40d8-a330-fc84c521a37d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:16:47.361075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.361041    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26772c06-318e-40d8-a330-fc84c521a37d-cert\") pod \"llmisvc-controller-manager-6466c554d8-vzg9h\" (UID: \"26772c06-318e-40d8-a330-fc84c521a37d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:16:47.361251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.361100    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znj98\" (UniqueName: \"kubernetes.io/projected/26772c06-318e-40d8-a330-fc84c521a37d-kube-api-access-znj98\") pod \"llmisvc-controller-manager-6466c554d8-vzg9h\" (UID: \"26772c06-318e-40d8-a330-fc84c521a37d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:16:47.363794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.363767    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26772c06-318e-40d8-a330-fc84c521a37d-cert\") pod \"llmisvc-controller-manager-6466c554d8-vzg9h\" (UID: \"26772c06-318e-40d8-a330-fc84c521a37d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:16:47.372914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.372890    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znj98\" (UniqueName: \"kubernetes.io/projected/26772c06-318e-40d8-a330-fc84c521a37d-kube-api-access-znj98\") pod \"llmisvc-controller-manager-6466c554d8-vzg9h\" (UID: \"26772c06-318e-40d8-a330-fc84c521a37d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:16:47.554726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.554608    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:16:47.853309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.853235    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"]
Oct 30 19:16:47.856770 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:16:47.856733    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26772c06_318e_40d8_a330_fc84c521a37d.slice/crio-56621f89a6b0fe16abafd21ec6582fd4065af9167504df0e1b2dfe5bc0a7f366 WatchSource:0}: Error finding container 56621f89a6b0fe16abafd21ec6582fd4065af9167504df0e1b2dfe5bc0a7f366: Status 404 returned error can't find the container with id 56621f89a6b0fe16abafd21ec6582fd4065af9167504df0e1b2dfe5bc0a7f366
Oct 30 19:16:47.858768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.858747    4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:16:47.953248 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.953211    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" event={"ID":"9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f","Type":"ContainerStarted","Data":"569818a1ff9358cfea2c754480a705498f4d9575e38f3f86d639eaec4b4c9b58"}
Oct 30 19:16:47.954411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:47.954385    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h" event={"ID":"26772c06-318e-40d8-a330-fc84c521a37d","Type":"ContainerStarted","Data":"56621f89a6b0fe16abafd21ec6582fd4065af9167504df0e1b2dfe5bc0a7f366"}
Oct 30 19:16:48.051814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:48.051783    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:16:48.052066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:48.052042    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cllrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:16:48.052152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:48.052096    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" podUID="9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 19:16:49.051893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:49.051860    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cllrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:16:49.052314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:49.051914    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" podUID="9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 19:16:49.820748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:49.820303    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:49.964570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:49.964537    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h" event={"ID":"26772c06-318e-40d8-a330-fc84c521a37d","Type":"ContainerStarted","Data":"ddf72717dc343dc2822204e40a0b71f223eef44f6305854fdbb4c43253834549"}
Oct 30 19:16:49.964773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:49.964632    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:16:49.982974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:49.982928    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h" podStartSLOduration=1.3653067700000001 podStartE2EDuration="2.982915085s" totalImagesPullingTime="1.617608315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:49.981799742 +0000 UTC m=+644.884812779" watchObservedRunningTime="2026-10-30 19:16:49.982915085 +0000 UTC m=+644.885928121"
Oct 30 19:16:49.994143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:49.994112    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:50.051980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:50.051944    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cllrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:16:50.052403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:50.051996    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" podUID="9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 19:16:51.051869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:51.051831    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cllrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:16:51.052076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:51.051890    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" podUID="9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 19:16:52.051457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:52.051413    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-cllrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:16:52.051629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:52.051484    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc" podUID="9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 30 19:16:52.145391 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:16:52.145359    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:16:53.055186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:53.055159    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:16:53.055453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:53.055436    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:16:53.056203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:53.056182    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-cllrc"
Oct 30 19:16:55.451393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:16:55.451356    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:10.330884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:10.330853    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:10.510812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:10.510758    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:20.971465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:20.971386    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vzg9h"
Oct 30 19:17:21.022733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.022680    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"]
Oct 30 19:17:21.022975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.022938    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8" podUID="a29a6635-9417-40b6-b136-bd7ed195848a" containerName="manager" containerID="cri-o://b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff" gracePeriod=30
Oct 30 19:17:21.276482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.276461    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:17:21.373274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.373249    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29a6635-9417-40b6-b136-bd7ed195848a-kube-api-access-4k9pb\" (UniqueName: \"kubernetes.io/projected/a29a6635-9417-40b6-b136-bd7ed195848a-kube-api-access-4k9pb\") pod \"a29a6635-9417-40b6-b136-bd7ed195848a\" (UID: \"a29a6635-9417-40b6-b136-bd7ed195848a\") "
Oct 30 19:17:21.373454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.373316    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29a6635-9417-40b6-b136-bd7ed195848a-cert\" (UniqueName: \"kubernetes.io/secret/a29a6635-9417-40b6-b136-bd7ed195848a-cert\") pod \"a29a6635-9417-40b6-b136-bd7ed195848a\" (UID: \"a29a6635-9417-40b6-b136-bd7ed195848a\") "
Oct 30 19:17:21.375625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.375594    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29a6635-9417-40b6-b136-bd7ed195848a-kube-api-access-4k9pb" pod "a29a6635-9417-40b6-b136-bd7ed195848a" (UID: "a29a6635-9417-40b6-b136-bd7ed195848a"). InnerVolumeSpecName "kube-api-access-4k9pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:21.375743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.375705    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29a6635-9417-40b6-b136-bd7ed195848a-cert" pod "a29a6635-9417-40b6-b136-bd7ed195848a" (UID: "a29a6635-9417-40b6-b136-bd7ed195848a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:21.474404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.474376    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k9pb\" (UniqueName: \"kubernetes.io/projected/a29a6635-9417-40b6-b136-bd7ed195848a-kube-api-access-4k9pb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:21.474404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.474399    4925 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a29a6635-9417-40b6-b136-bd7ed195848a-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:21.994530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:21.994492    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:22.109774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.109745    4925 generic.go:356] "Generic (PLEG): container finished" podID="a29a6635-9417-40b6-b136-bd7ed195848a" containerID="b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff" exitCode=0
Oct 30 19:17:22.109926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.109794    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"
Oct 30 19:17:22.109926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.109825    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8" event={"ID":"a29a6635-9417-40b6-b136-bd7ed195848a","Type":"ContainerDied","Data":"b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff"}
Oct 30 19:17:22.109926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.109882    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8" event={"ID":"a29a6635-9417-40b6-b136-bd7ed195848a","Type":"ContainerDied","Data":"3c2220f5780980528aaf965e7355599a73e856e4dd251d89416661833d87038a"}
Oct 30 19:17:22.109926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.109905    4925 scope.go:122] "RemoveContainer" containerID="b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff"
Oct 30 19:17:22.118865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.118845    4925 scope.go:122] "RemoveContainer" containerID="b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff"
Oct 30 19:17:22.119237 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:17:22.119193    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff\": container with ID starting with b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff not found: ID does not exist" containerID="b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff"
Oct 30 19:17:22.119313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.119238    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff"} err="failed to get container status \"b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff\": rpc error: code = NotFound desc = could not find container \"b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff\": container with ID starting with b83df41ba60c3c30cf5c192caf0e0f1fbcd5f5ded901f61d307ae799c0bbc8ff not found: ID does not exist"
Oct 30 19:17:22.125664 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:17:22.125641    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:17:22.129570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.129551    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"]
Oct 30 19:17:22.133325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.133308    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks9m8"]
Oct 30 19:17:22.207027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:22.206999    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:23.260708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:23.260654    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29a6635-9417-40b6-b136-bd7ed195848a" path="/var/lib/kubelet/pods/a29a6635-9417-40b6-b136-bd7ed195848a/volumes"
Oct 30 19:17:24.277516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:24.277484    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:24.514231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:24.514194    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:41.438796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:41.438760    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:43.046182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:43.046154    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:44.231636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:44.231591    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:45.884655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:45.884617    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:46.772318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:46.772286    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:47.794008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:47.793970    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:49.185114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:49.185072    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:50.609020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:50.608979    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:52.177122 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:17:52.177071    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:17:55.506157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:17:55.506118    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:05.042031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.041997    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fdffc46cf-sq9nd"]
Oct 30 19:18:05.042419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.042166    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd" podUID="cb66ba6d-857a-4a36-a920-c874426bba28" containerName="authorino" containerID="cri-o://6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2" gracePeriod=30
Oct 30 19:18:05.289305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.289278    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:18:05.319924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.319844    4925 generic.go:356] "Generic (PLEG): container finished" podID="cb66ba6d-857a-4a36-a920-c874426bba28" containerID="6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2" exitCode=0
Oct 30 19:18:05.319924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.319889    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd"
Oct 30 19:18:05.320077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.319927    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd" event={"ID":"cb66ba6d-857a-4a36-a920-c874426bba28","Type":"ContainerDied","Data":"6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2"}
Oct 30 19:18:05.320077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.319968    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fdffc46cf-sq9nd" event={"ID":"cb66ba6d-857a-4a36-a920-c874426bba28","Type":"ContainerDied","Data":"c58acb5f3e265d803d88f37f54490c206f52730102bfc633c574d07f5a7b5b2a"}
Oct 30 19:18:05.320077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.319989    4925 scope.go:122] "RemoveContainer" containerID="6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2"
Oct 30 19:18:05.329401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.329377    4925 scope.go:122] "RemoveContainer" containerID="6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2"
Oct 30 19:18:05.329772 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:18:05.329671    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2\": container with ID starting with 6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2 not found: ID does not exist" containerID="6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2"
Oct 30 19:18:05.329772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.329711    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2"} err="failed to get container status \"6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2\": rpc error: code = NotFound desc = could not find container \"6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2\": container with ID starting with 6e4877ed0a6d2f619465b7af0736bb9e82ed36dd34bb26b4361e55bb3e7a53b2 not found: ID does not exist"
Oct 30 19:18:05.454934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.454904    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb66ba6d-857a-4a36-a920-c874426bba28-tls-cert\" (UniqueName: \"kubernetes.io/secret/cb66ba6d-857a-4a36-a920-c874426bba28-tls-cert\") pod \"cb66ba6d-857a-4a36-a920-c874426bba28\" (UID: \"cb66ba6d-857a-4a36-a920-c874426bba28\") "
Oct 30 19:18:05.455075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.454946    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb66ba6d-857a-4a36-a920-c874426bba28-kube-api-access-2njcr\" (UniqueName: \"kubernetes.io/projected/cb66ba6d-857a-4a36-a920-c874426bba28-kube-api-access-2njcr\") pod \"cb66ba6d-857a-4a36-a920-c874426bba28\" (UID: \"cb66ba6d-857a-4a36-a920-c874426bba28\") "
Oct 30 19:18:05.457078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.457049    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb66ba6d-857a-4a36-a920-c874426bba28-kube-api-access-2njcr" pod "cb66ba6d-857a-4a36-a920-c874426bba28" (UID: "cb66ba6d-857a-4a36-a920-c874426bba28"). InnerVolumeSpecName "kube-api-access-2njcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:05.465012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.464992    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb66ba6d-857a-4a36-a920-c874426bba28-tls-cert" pod "cb66ba6d-857a-4a36-a920-c874426bba28" (UID: "cb66ba6d-857a-4a36-a920-c874426bba28"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:05.555790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.555759    4925 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb66ba6d-857a-4a36-a920-c874426bba28-tls-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:05.555790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.555788    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2njcr\" (UniqueName: \"kubernetes.io/projected/cb66ba6d-857a-4a36-a920-c874426bba28-kube-api-access-2njcr\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:05.644121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.644094    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fdffc46cf-sq9nd"]
Oct 30 19:18:05.651114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.651093    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6fdffc46cf-sq9nd"]
Oct 30 19:18:05.690167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.690141    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:05.855924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:05.855887    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:07.261743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:07.261711    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb66ba6d-857a-4a36-a920-c874426bba28" path="/var/lib/kubelet/pods/cb66ba6d-857a-4a36-a920-c874426bba28/volumes"
Oct 30 19:18:11.502386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:11.502335    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:11.673942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:11.673907    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:20.216713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:20.216658    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:20.380519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:20.380485    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:22.123876 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:18:22.123837    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:18:22.448137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.448109    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"]
Oct 30 19:18:22.448465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.448451    4925 cpu_manager.go:414] "containerMap: removing container" podUID="cb66ba6d-857a-4a36-a920-c874426bba28" containerName="authorino"
Oct 30 19:18:22.448519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.448466    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb66ba6d-857a-4a36-a920-c874426bba28" containerName="authorino"
Oct 30 19:18:22.448519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.448484    4925 cpu_manager.go:414] "containerMap: removing container" podUID="a29a6635-9417-40b6-b136-bd7ed195848a" containerName="manager"
Oct 30 19:18:22.448519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.448489    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29a6635-9417-40b6-b136-bd7ed195848a" containerName="manager"
Oct 30 19:18:22.448619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.448564    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb66ba6d-857a-4a36-a920-c874426bba28" containerName="authorino"
Oct 30 19:18:22.448619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.448572    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29a6635-9417-40b6-b136-bd7ed195848a" containerName="manager"
Oct 30 19:18:22.451398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.451377    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.467291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.467266    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"]
Oct 30 19:18:22.595769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.595733    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.595769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.595770    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.595982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.595796    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.595982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.595855    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.595982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.595961    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.596090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.596007    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qjbm\" (UniqueName: \"kubernetes.io/projected/68707443-c353-4c78-9d2f-fb9e915c6b15-kube-api-access-8qjbm\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.596090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.596073    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.596152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.596111    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.596243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.596228    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68707443-c353-4c78-9d2f-fb9e915c6b15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.696757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.696726    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68707443-c353-4c78-9d2f-fb9e915c6b15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.696920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.696779    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.696920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.696803    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.696920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.696827    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.696920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.696848    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.697135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.696974    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.697135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.697008    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qjbm\" (UniqueName: \"kubernetes.io/projected/68707443-c353-4c78-9d2f-fb9e915c6b15-kube-api-access-8qjbm\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.697135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.697057    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.697135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.697097    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.697316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.697196    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.697316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.697240    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.697425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.697368    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.697491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.697473    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68707443-c353-4c78-9d2f-fb9e915c6b15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.697541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.697486    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.699408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.699363    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.699616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.699600    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.708273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.708249    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qjbm\" (UniqueName: \"kubernetes.io/projected/68707443-c353-4c78-9d2f-fb9e915c6b15-kube-api-access-8qjbm\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.708412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.708392    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68707443-c353-4c78-9d2f-fb9e915c6b15-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-hcvdj\" (UID: \"68707443-c353-4c78-9d2f-fb9e915c6b15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:22.769924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:22.769893    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:23.055458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:23.055325    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"]
Oct 30 19:18:23.058106 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:18:23.058080    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68707443_c353_4c78_9d2f_fb9e915c6b15.slice/crio-8560aff2f5a685099318b4e7eff0b7285206c720b8ca43ba8946bb1154c2e12e WatchSource:0}: Error finding container 8560aff2f5a685099318b4e7eff0b7285206c720b8ca43ba8946bb1154c2e12e: Status 404 returned error can't find the container with id 8560aff2f5a685099318b4e7eff0b7285206c720b8ca43ba8946bb1154c2e12e
Oct 30 19:18:23.060351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:23.060312    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:18:23.060450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:23.060381    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:18:23.060450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:23.060416    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:18:23.398524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:23.398492    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" event={"ID":"68707443-c353-4c78-9d2f-fb9e915c6b15","Type":"ContainerStarted","Data":"b42a42504e10236c5e67feadce08eeaf692e3c8f7f4675ac1ea9058934f2e193"}
Oct 30 19:18:23.398906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:23.398546    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" event={"ID":"68707443-c353-4c78-9d2f-fb9e915c6b15","Type":"ContainerStarted","Data":"8560aff2f5a685099318b4e7eff0b7285206c720b8ca43ba8946bb1154c2e12e"}
Oct 30 19:18:23.422638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:23.421557    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podStartSLOduration=1.421535574 podStartE2EDuration="1.421535574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:23.419511478 +0000 UTC m=+738.322524511" watchObservedRunningTime="2026-10-30 19:18:23.421535574 +0000 UTC m=+738.324548612"
Oct 30 19:18:23.770298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:23.770218    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:24.772012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:24.771977    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-hcvdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:18:24.772399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:24.772028    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podUID="68707443-c353-4c78-9d2f-fb9e915c6b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 19:18:25.770927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:25.770897    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-hcvdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:18:25.771113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:25.770948    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podUID="68707443-c353-4c78-9d2f-fb9e915c6b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 19:18:26.770820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:26.770788    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-hcvdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:18:26.771212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:26.770836    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podUID="68707443-c353-4c78-9d2f-fb9e915c6b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 19:18:27.770552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:27.770518    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-hcvdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:18:27.770728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:27.770586    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podUID="68707443-c353-4c78-9d2f-fb9e915c6b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 19:18:28.770389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:28.770354    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-hcvdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:18:28.770815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:28.770420    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podUID="68707443-c353-4c78-9d2f-fb9e915c6b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 19:18:29.770702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:29.770665    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-hcvdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:18:29.771075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:29.770746    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podUID="68707443-c353-4c78-9d2f-fb9e915c6b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 19:18:31.232300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:31.232270    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:31.232787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:31.232555    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:31.233549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:31.233531    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj"
Oct 30 19:18:37.637928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:37.637894    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:38.165524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:38.165490    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:39.915678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:39.915633    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:40.298905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:40.298807    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:41.484851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:41.484816    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:41.658895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:41.658863    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:52.125122 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:18:52.125038    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:18:55.565069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:18:55.565033    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:03.907646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:03.907613    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:04.101375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.101339    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:04.487867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.487831    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:04.546025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.545985    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"]
Oct 30 19:19:04.549500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.549479    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.560118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.560098    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-bce71f-openshift-default-dockercfg-w6thn\""
Oct 30 19:19:04.564507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.564480    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"]
Oct 30 19:19:04.671402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.671349    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25918327-0d30-438b-979c-4531f0fd270f-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.671402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.671402    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.671647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.671430    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.671647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.671548    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.671647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.671615    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25918327-0d30-438b-979c-4531f0fd270f-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.671830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.671722    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.671830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.671755    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.671830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.671794    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q82z\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-kube-api-access-9q82z\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.671986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.671903    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.692971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.692935    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:04.743972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.743896    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"]
Oct 30 19:19:04.747347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.747319    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.760220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.760199    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2d5d53-openshift-default-dockercfg-mbh26\""
Oct 30 19:19:04.763787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.763749    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"]
Oct 30 19:19:04.773124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773098    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q82z\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-kube-api-access-9q82z\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773149    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773183    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25918327-0d30-438b-979c-4531f0fd270f-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773209    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773237    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773261    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773310    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25918327-0d30-438b-979c-4531f0fd270f-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773353    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773388    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773888    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.773990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.773677    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.774068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.774048    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.774130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.774073    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.774207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.774184    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25918327-0d30-438b-979c-4531f0fd270f-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.776252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.776233    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.776382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.776363    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25918327-0d30-438b-979c-4531f0fd270f-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.796923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.796888    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q82z\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-kube-api-access-9q82z\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.797315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.797290    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.863146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.863115    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:04.874428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.874402    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8411f4a9-165d-47a5-a255-e6d07376731e-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.874528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.874443    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.874528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.874473    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.874641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.874545    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.874641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.874570    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.874641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.874638    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.874816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.874658    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn8sz\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-kube-api-access-gn8sz\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.874816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.874718    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8411f4a9-165d-47a5-a255-e6d07376731e-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.874816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.874753    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.884928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.884902    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:04.949542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.943336    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"]
Oct 30 19:19:04.949542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.948076    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:04.960761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.960653    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-39b867-openshift-default-dockercfg-8z9sp\""
Oct 30 19:19:04.967131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.967109    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"]
Oct 30 19:19:04.975139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975119    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975148    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn8sz\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-kube-api-access-gn8sz\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975174    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8411f4a9-165d-47a5-a255-e6d07376731e-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975222    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975274    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8411f4a9-165d-47a5-a255-e6d07376731e-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975300    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975333    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975360    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975390    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975802    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.975948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975928    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.976027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.975943    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.976119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.976019    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8411f4a9-165d-47a5-a255-e6d07376731e-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.976119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.976058    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.978246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.978216    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8411f4a9-165d-47a5-a255-e6d07376731e-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.978340 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.978305    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.987301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.987281    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:04.987413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:04.987344    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn8sz\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-kube-api-access-gn8sz\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:05.057513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.057430    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:05.064355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.064330    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:05.076071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.076042    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.076184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.076083    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.076245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.076186    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.076245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.076220    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.076360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.076248    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m268\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-kube-api-access-5m268\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.076444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.076419    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.076502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.076483    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.076682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.076569    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.076682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.076608    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.181998    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.182067    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.182105    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.182141    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.182165    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.182214    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.182247    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.182271    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m268\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-kube-api-access-5m268\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.182356    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.183106    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.183377    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.184188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.183989    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.185114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.184253    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.185114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.184469    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.187596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.187410    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.187819 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:19:05.187800    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25918327_0d30_438b_979c_4531f0fd270f.slice/crio-95a4da07604aaa7b0f087970763258055ad443334f9dd89c748cafe4632818da WatchSource:0}: Error finding container 95a4da07604aaa7b0f087970763258055ad443334f9dd89c748cafe4632818da: Status 404 returned error can't find the container with id 95a4da07604aaa7b0f087970763258055ad443334f9dd89c748cafe4632818da
Oct 30 19:19:05.188199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.188180    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.189323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.189300    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"]
Oct 30 19:19:05.190125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.190094    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:05.190203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.190153    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:05.190203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.190182    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:05.199004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.198965    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m268\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-kube-api-access-5m268\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.200051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.200018    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.240813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.240763    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:05.267421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.267391    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-39b867-openshift-default-dockercfg-8z9sp\""
Oct 30 19:19:05.275933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.275905    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:05.371465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.369864    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"]
Oct 30 19:19:05.373373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.373329    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:05.373474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.373413    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:05.373474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.373454    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:05.412323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.412268    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:05.585649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.585616    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"]
Oct 30 19:19:05.589011 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:19:05.588987    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c5d294_364e_42fe_85dc_dce17f1a31ea.slice/crio-02b56bc9e00f17a006a74e7a32ac44d087823512b2956d215e8ae9dc1c46f530 WatchSource:0}: Error finding container 02b56bc9e00f17a006a74e7a32ac44d087823512b2956d215e8ae9dc1c46f530: Status 404 returned error can't find the container with id 02b56bc9e00f17a006a74e7a32ac44d087823512b2956d215e8ae9dc1c46f530
Oct 30 19:19:05.591792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.591743    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:05.591882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.591833    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:05.591924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.591879    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:05.623124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.623093    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" event={"ID":"f2c5d294-364e-42fe-85dc-dce17f1a31ea","Type":"ContainerStarted","Data":"02b56bc9e00f17a006a74e7a32ac44d087823512b2956d215e8ae9dc1c46f530"}
Oct 30 19:19:05.625428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.624887    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" event={"ID":"8411f4a9-165d-47a5-a255-e6d07376731e","Type":"ContainerStarted","Data":"105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa"}
Oct 30 19:19:05.625428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.624936    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" event={"ID":"8411f4a9-165d-47a5-a255-e6d07376731e","Type":"ContainerStarted","Data":"9880e509c4052ae9ea646cf77dd6154ac7702183a911bb51365b8ea6cb6ae12e"}
Oct 30 19:19:05.627293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.626836    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" event={"ID":"25918327-0d30-438b-979c-4531f0fd270f","Type":"ContainerStarted","Data":"a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5"}
Oct 30 19:19:05.627293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.626875    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" event={"ID":"25918327-0d30-438b-979c-4531f0fd270f","Type":"ContainerStarted","Data":"95a4da07604aaa7b0f087970763258055ad443334f9dd89c748cafe4632818da"}
Oct 30 19:19:05.649054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.649006    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" podStartSLOduration=1.648989737 podStartE2EDuration="1.648989737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:05.648867734 +0000 UTC m=+780.551880783" watchObservedRunningTime="2026-10-30 19:19:05.648989737 +0000 UTC m=+780.552002774"
Oct 30 19:19:05.673090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.672890    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" podStartSLOduration=1.672872213 podStartE2EDuration="1.672872213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:05.671733285 +0000 UTC m=+780.574746318" watchObservedRunningTime="2026-10-30 19:19:05.672872213 +0000 UTC m=+780.575885251"
Oct 30 19:19:05.864229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:05.864142    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:06.065025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:06.064993    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:06.070751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:06.070728    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:06.634274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:06.634238    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" event={"ID":"f2c5d294-364e-42fe-85dc-dce17f1a31ea","Type":"ContainerStarted","Data":"c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e"}
Oct 30 19:19:06.634708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:06.634673    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:06.635386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:06.635364    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:06.663149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:06.663108    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" podStartSLOduration=2.663095836 podStartE2EDuration="2.663095836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:06.661495514 +0000 UTC m=+781.564508552" watchObservedRunningTime="2026-10-30 19:19:06.663095836 +0000 UTC m=+781.566108936"
Oct 30 19:19:06.869064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:06.869029    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:07.276625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:07.276595    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:07.281356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:07.281336    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:07.640524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:07.640497    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:07.640787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:07.640768    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:07.641399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:07.641381    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:07.641621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:07.641605    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:08.027627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:08.027537    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:08.208548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:08.208509    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:08.429799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:08.429768    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:08.611441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:08.611410    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:08.802627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:08.802547    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:08.985038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:08.985009    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:09.454045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:09.454014    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:09.616934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:09.616901    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:10.144730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:10.144672    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:10.307053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:10.307020    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:10.464294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:10.464206    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:11.105491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:11.105463    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:11.275461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:11.275426    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:11.583615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:11.583585    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:11.739840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:11.739804    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:12.059085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:12.059056    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:12.263383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:12.263351    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:12.489434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:12.489400    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:12.722621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:12.722590    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:12.895903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:12.895869    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:13.111426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.111393    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:13.273717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.273611    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:13.431540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.431510    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:13.718542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.718509    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"]
Oct 30 19:19:13.723227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.723204    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:13.731919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.731899    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-39b867\""
Oct 30 19:19:13.734280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.734257    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"]
Oct 30 19:19:13.755120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.755089    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:13.867262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.867233    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8\" (UID: \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:13.867407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.867286    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxmsh\" (UniqueName: \"kubernetes.io/projected/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-kube-api-access-dxmsh\") pod \"payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8\" (UID: \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:13.934287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.934244    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:13.968410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.968383    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8\" (UID: \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:13.968529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.968436    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxmsh\" (UniqueName: \"kubernetes.io/projected/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-kube-api-access-dxmsh\") pod \"payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8\" (UID: \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:13.969127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.969087    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8\" (UID: \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:13.981420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:13.981393    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxmsh\" (UniqueName: \"kubernetes.io/projected/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-kube-api-access-dxmsh\") pod \"payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8\" (UID: \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:14.037951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:14.037920    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:14.099163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:14.099077    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:14.321203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:14.318464    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"]
Oct 30 19:19:14.325366 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:19:14.325337    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ee03ca_995d_4c4a_85c2_f7b28a1e40f1.slice/crio-8ef60d69a16bbff1a2400217ccaef5b78cd56c10e49d779aa7d6592a419d83e8 WatchSource:0}: Error finding container 8ef60d69a16bbff1a2400217ccaef5b78cd56c10e49d779aa7d6592a419d83e8: Status 404 returned error can't find the container with id 8ef60d69a16bbff1a2400217ccaef5b78cd56c10e49d779aa7d6592a419d83e8
Oct 30 19:19:14.675268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:14.675234    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8" event={"ID":"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1","Type":"ContainerStarted","Data":"8ef60d69a16bbff1a2400217ccaef5b78cd56c10e49d779aa7d6592a419d83e8"}
Oct 30 19:19:14.887925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:14.887888    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:15.743304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:15.743267    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:15.906537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:15.906507    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:16.252985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.252884    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:16.686275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.686243    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8" event={"ID":"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1","Type":"ContainerStarted","Data":"c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1"}
Oct 30 19:19:16.686473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.686320    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:16.706723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.706649    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8" podStartSLOduration=1.510159792 podStartE2EDuration="3.706631076s" totalImagesPullingTime="2.196471284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:16.705913944 +0000 UTC m=+791.608926981" watchObservedRunningTime="2026-10-30 19:19:16.706631076 +0000 UTC m=+791.609644117"
Oct 30 19:19:16.927161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.927132    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"]
Oct 30 19:19:16.930766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.930744    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:16.935804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.935777    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-bce71f-serving-cert\""
Oct 30 19:19:16.935804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.935780    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:19:16.935995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.935937    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f8msk\""
Oct 30 19:19:16.936265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.936239    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:19:16.936376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.936339    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:19:16.942800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:16.941332    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"]
Oct 30 19:19:17.083096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.083061    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:17.120265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.120224    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ac07f1-44d5-427c-9e98-7933ef9d0abd-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.120463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.120273    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4z44\" (UniqueName: \"kubernetes.io/projected/36ac07f1-44d5-427c-9e98-7933ef9d0abd-kube-api-access-l4z44\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.120463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.120407    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-metrics-certs\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.120582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.120461    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.221316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.221214    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ac07f1-44d5-427c-9e98-7933ef9d0abd-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.221316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.221257    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4z44\" (UniqueName: \"kubernetes.io/projected/36ac07f1-44d5-427c-9e98-7933ef9d0abd-kube-api-access-l4z44\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.221316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.221299    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-metrics-certs\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.221579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.221336    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.221994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.221968    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ac07f1-44d5-427c-9e98-7933ef9d0abd-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.224002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.223978    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-metrics-certs\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.224169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.224146    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.264056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.264029    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4z44\" (UniqueName: \"kubernetes.io/projected/36ac07f1-44d5-427c-9e98-7933ef9d0abd-kube-api-access-l4z44\") pod \"maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.419715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.419268    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:17.546918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.546842    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:17.847527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:17.847456    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"]
Oct 30 19:19:17.850993 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:19:17.850966    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ac07f1_44d5_427c_9e98_7933ef9d0abd.slice/crio-a1cefc73b4d72dcf075225a2235161a75b1dd6eac2324ec7b5d3c1f770d10350 WatchSource:0}: Error finding container a1cefc73b4d72dcf075225a2235161a75b1dd6eac2324ec7b5d3c1f770d10350: Status 404 returned error can't find the container with id a1cefc73b4d72dcf075225a2235161a75b1dd6eac2324ec7b5d3c1f770d10350
Oct 30 19:19:18.022415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:18.022337    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:18.194444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:18.194416    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:18.520965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:18.520884    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:18.745580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:18.745540    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9" event={"ID":"36ac07f1-44d5-427c-9e98-7933ef9d0abd","Type":"ContainerStarted","Data":"a1cefc73b4d72dcf075225a2235161a75b1dd6eac2324ec7b5d3c1f770d10350"}
Oct 30 19:19:18.958768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:18.958670    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:19.866834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:19.866728    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:20.619713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:20.619211    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:21.369202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:21.369115    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:21.762794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:21.762756    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9" event={"ID":"36ac07f1-44d5-427c-9e98-7933ef9d0abd","Type":"ContainerStarted","Data":"f7274807097589218639f409ad55741d325704a3335360dcbc791f2dde4f6919"}
Oct 30 19:19:21.762990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:21.762905    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:21.787026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:21.786968    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9" podStartSLOduration=2.561543193 podStartE2EDuration="5.7869508s" totalImagesPullingTime="3.225407607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:21.786434326 +0000 UTC m=+796.689447365" watchObservedRunningTime="2026-10-30 19:19:21.7869508 +0000 UTC m=+796.689963838"
Oct 30 19:19:22.143710 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:19:22.143660    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:19:22.365507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:22.365472    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:23.103715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:23.099185    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:23.305712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:23.303611    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:23.689998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:23.689854    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:19:23.689998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:23.689930    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:19:23.771322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:23.771194    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:23.951475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:23.951326    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:24.208255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.208130    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:24.812585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.812547    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"]
Oct 30 19:19:24.857105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.857075    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"]
Oct 30 19:19:24.857396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.857382    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:24.871191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.869664    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"kube-root-ca.crt\""
Oct 30 19:19:24.871191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.869803    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"openshift-service-ca.crt\""
Oct 30 19:19:24.871191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.870436    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"default-dockercfg-fq74v\""
Oct 30 19:19:24.871191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.870556    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"simulator-w1-cfa4bb-kserve-self-signed-certs\""
Oct 30 19:19:24.928725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.928630    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:24.962124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.962084    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tls-certs\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:24.962124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.962126    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-home\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:24.962364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.962155    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8866f\" (UniqueName: \"kubernetes.io/projected/f3cda07a-70d6-468f-8c34-1ce0c64f7653-kube-api-access-8866f\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:24.962364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.962173    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tmp-dir\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:24.962364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.962190    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-dshm\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:24.962364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:24.962267    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-model-cache\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.064003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.063247    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-model-cache\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.064003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.063297    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tls-certs\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.064003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.063332    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-home\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.064003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.063373    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8866f\" (UniqueName: \"kubernetes.io/projected/f3cda07a-70d6-468f-8c34-1ce0c64f7653-kube-api-access-8866f\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.064003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.063400    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tmp-dir\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.064003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.063425    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-dshm\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.064889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.064863    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tmp-dir\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.065237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.065217    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-model-cache\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.065920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.065503    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-home\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.066929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.066902    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-dshm\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.072278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.072254    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tls-certs\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.077033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.077006    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8866f\" (UniqueName: \"kubernetes.io/projected/f3cda07a-70d6-468f-8c34-1ce0c64f7653-kube-api-access-8866f\") pod \"simulator-w1-cfa4bb-kserve-7776c645b-qhhpt\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") " pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.136620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.136548    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:25.177644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.177607    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:25.549712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.542799    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"]
Oct 30 19:19:25.551194 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:19:25.551011    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3cda07a_70d6_468f_8c34_1ce0c64f7653.slice/crio-a0f997d9e9ac4835d92c8e9458832dd09c62c5f08d20f42bec0fb789facc3117 WatchSource:0}: Error finding container a0f997d9e9ac4835d92c8e9458832dd09c62c5f08d20f42bec0fb789facc3117: Status 404 returned error can't find the container with id a0f997d9e9ac4835d92c8e9458832dd09c62c5f08d20f42bec0fb789facc3117
Oct 30 19:19:25.613721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.605131    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:25.772026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.771991    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:25.784551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:25.784528    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" event={"ID":"f3cda07a-70d6-468f-8c34-1ce0c64f7653","Type":"ContainerStarted","Data":"a0f997d9e9ac4835d92c8e9458832dd09c62c5f08d20f42bec0fb789facc3117"}
Oct 30 19:19:26.487476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:26.487438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:27.528318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:27.528273    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:27.694646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:27.694621    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:19:27.779774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:27.779585    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:19:28.076444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.076289    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:28.428349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.428321    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:28.720590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.720549    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:28.824797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.824761    4925 generic.go:356] "Generic (PLEG): container finished" podID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerID="c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e" exitCode=0
Oct 30 19:19:28.824990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.824968    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" event={"ID":"f2c5d294-364e-42fe-85dc-dce17f1a31ea","Type":"ContainerDied","Data":"c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e"}
Oct 30 19:19:28.825430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.825403    4925 scope.go:122] "RemoveContainer" containerID="c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e"
Oct 30 19:19:28.827484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.826879    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:28.827484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.827093    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:28.827484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.827149    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:28.827484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.827270    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" event={"ID":"f3cda07a-70d6-468f-8c34-1ce0c64f7653","Type":"ContainerStarted","Data":"156327d94c00c0a955f24f2678a3f5da15609d7bb91daf53d1c809bf8bf2e6b9"}
Oct 30 19:19:28.879656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.879604    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podStartSLOduration=2.152738465 podStartE2EDuration="4.879589227s" totalImagesPullingTime="2.726850762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:28.878985704 +0000 UTC m=+803.781998742" watchObservedRunningTime="2026-10-30 19:19:28.879589227 +0000 UTC m=+803.782602266"
Oct 30 19:19:28.905002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:28.904975    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:29.178292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:29.178256    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:29.327939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:29.327903    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:29.553711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:29.553608    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:29.838261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:29.838042    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" event={"ID":"f2c5d294-364e-42fe-85dc-dce17f1a31ea","Type":"ContainerStarted","Data":"92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881"}
Oct 30 19:19:29.915713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:29.914390    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:30.098924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:30.098796    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:30.276567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:30.276516    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:30.276869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:30.276842    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:19:30.276951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:30.276906    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 19:19:30.702572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:30.702498    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:31.277164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:31.277125    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:19:31.277729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:31.277186    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 19:19:31.638509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:31.638425    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:32.277277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.277239    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:19:32.278644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.277295    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 19:19:32.858386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.858348    4925 generic.go:356] "Generic (PLEG): container finished" podID="8411f4a9-165d-47a5-a255-e6d07376731e" containerID="105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa" exitCode=0
Oct 30 19:19:32.858591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.858426    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" event={"ID":"8411f4a9-165d-47a5-a255-e6d07376731e","Type":"ContainerDied","Data":"105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa"}
Oct 30 19:19:32.858993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.858965    4925 scope.go:122] "RemoveContainer" containerID="105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa"
Oct 30 19:19:32.860352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.860321    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:32.860459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.860354    4925 generic.go:356] "Generic (PLEG): container finished" podID="25918327-0d30-438b-979c-4531f0fd270f" containerID="a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5" exitCode=0
Oct 30 19:19:32.860459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.860388    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:32.860459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.860422    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:32.860577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.860419    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" event={"ID":"25918327-0d30-438b-979c-4531f0fd270f","Type":"ContainerDied","Data":"a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5"}
Oct 30 19:19:32.860799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.860782    4925 scope.go:122] "RemoveContainer" containerID="a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5"
Oct 30 19:19:32.862100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.862055    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:32.862206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.862131    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:32.862206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.862178    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:19:32.902712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:32.900898    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:33.235715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.235591    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"]
Oct 30 19:19:33.245208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.245184    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.245967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.245873    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"]
Oct 30 19:19:33.253582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.253553    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-27b5d5\"/\"kube-root-ca.crt\""
Oct 30 19:19:33.254197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.254064    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-27b5d5\"/\"openshift-service-ca.crt\""
Oct 30 19:19:33.255096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.254493    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-27b5d5\"/\"simulator-w4-27b5d5-kserve-self-signed-certs\""
Oct 30 19:19:33.255983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.255965    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-27b5d5\"/\"default-dockercfg-x4446\""
Oct 30 19:19:33.276598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.276556    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:19:33.276777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.276611    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 19:19:33.290064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.290032    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:33.365075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.364219    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-tmp-dir\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.365075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.364349    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f27b3d-570a-448e-8926-ed6643d8903b-tls-certs\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.365075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.364374    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-model-cache\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.365075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.364453    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-228kq\" (UniqueName: \"kubernetes.io/projected/e9f27b3d-570a-448e-8926-ed6643d8903b-kube-api-access-228kq\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.365075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.364516    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-home\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.365075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.364538    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-dshm\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.466747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.466274    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-228kq\" (UniqueName: \"kubernetes.io/projected/e9f27b3d-570a-448e-8926-ed6643d8903b-kube-api-access-228kq\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.466747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.466377    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-home\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.466747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.466414    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-dshm\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.466747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.466473    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-tmp-dir\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.466747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.466512    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f27b3d-570a-448e-8926-ed6643d8903b-tls-certs\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.466747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.466535    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-model-cache\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.467557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.467423    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-home\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.467557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.467543    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-model-cache\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.467851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.467830    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-tmp-dir\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.470413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.470368    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-dshm\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.471078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.471055    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f27b3d-570a-448e-8926-ed6643d8903b-tls-certs\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.493962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.493886    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-228kq\" (UniqueName: \"kubernetes.io/projected/e9f27b3d-570a-448e-8926-ed6643d8903b-kube-api-access-228kq\") pod \"simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") " pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.571176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.570883    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:33.587812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.587784    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:33.770898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.770781    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:33.909958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.908029    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" event={"ID":"8411f4a9-165d-47a5-a255-e6d07376731e","Type":"ContainerStarted","Data":"b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853"}
Oct 30 19:19:33.962594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:33.961814    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" event={"ID":"25918327-0d30-438b-979c-4531f0fd270f","Type":"ContainerStarted","Data":"7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224"}
Oct 30 19:19:34.087730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.087626    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:34.088887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.088764    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:34.088887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.088830    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"]
Oct 30 19:19:34.090263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.090240    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:34.090438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.090413    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:19:34.285636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.281767    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:19:34.285636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.281824    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 19:19:34.288131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.287577    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:34.475712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.472530    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:34.843237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.843202    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:34.876256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.874581    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:34.876256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.874805    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:34.876256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.874867    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 19:19:34.973733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.973676    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" event={"ID":"e9f27b3d-570a-448e-8926-ed6643d8903b","Type":"ContainerStarted","Data":"586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa"}
Oct 30 19:19:34.973733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:34.973738    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" event={"ID":"e9f27b3d-570a-448e-8926-ed6643d8903b","Type":"ContainerStarted","Data":"c506c9691b901ac70f01076c23f983ffe4a894ee9c6264f0d602199b7d49f0fb"}
Oct 30 19:19:35.023444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.023371    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podStartSLOduration=2.023352674 podStartE2EDuration="2.023352674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:35.022394114 +0000 UTC m=+809.925407152" watchObservedRunningTime="2026-10-30 19:19:35.023352674 +0000 UTC m=+809.926365714"
Oct 30 19:19:35.056390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.056191    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:35.065308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.065277    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:35.065470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.065325    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:19:35.178825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.178795    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:35.225938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.225908    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:35.382760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.382730    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:35.383222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.383196    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:35.384474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.384453    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:19:35.393167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.393142    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:35.571624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.571549    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:35.863601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.863521    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:35.863601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.863574    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 19:19:35.991082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:35.991051    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:19:36.065506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:36.065476    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:36.065806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:36.065765    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:19:36.159717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:36.159652    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:36.638767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:36.637792    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:36.864117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:36.864075    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:36.864288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:36.864133    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 19:19:36.923711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:36.920066    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:37.064937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.064906    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:37.065207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.065179    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:19:37.446713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.444611    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"]
Oct 30 19:19:37.526715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.526486    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"]
Oct 30 19:19:37.526715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.526608    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.548712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.546558    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39b867\"/\"default-dockercfg-v6xnn\""
Oct 30 19:19:37.564086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.562042    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-39b867\"/\"kube-root-ca.crt\""
Oct 30 19:19:37.564086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.562165    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-39b867\"/\"openshift-service-ca.crt\""
Oct 30 19:19:37.576940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.575096    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39b867\"/\"simulator-w2-39b867-kserve-self-signed-certs\""
Oct 30 19:19:37.576940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.575560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:37.586105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.586082    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66623a2e-1062-4b84-99fe-7af3b038c072-tls-certs\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.586279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.586264    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-home\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.586421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.586410    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-model-cache\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.586532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.586518    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvg6g\" (UniqueName: \"kubernetes.io/projected/66623a2e-1062-4b84-99fe-7af3b038c072-kube-api-access-mvg6g\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.586664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.586654    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-dshm\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.586795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.586781    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-tmp-dir\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.649313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.647783    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:37.687862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.687831    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvg6g\" (UniqueName: \"kubernetes.io/projected/66623a2e-1062-4b84-99fe-7af3b038c072-kube-api-access-mvg6g\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.688068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.687916    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-dshm\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.688068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.687964    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-tmp-dir\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.688068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.687996    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66623a2e-1062-4b84-99fe-7af3b038c072-tls-certs\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.688068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.688016    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-home\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.688278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.688074    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-model-cache\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.688436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.688419    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-model-cache\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.689022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.689004    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-tmp-dir\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.689301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.689257    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-home\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.692233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.692209    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66623a2e-1062-4b84-99fe-7af3b038c072-tls-certs\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.692417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.691653    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-dshm\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.724214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.722423    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvg6g\" (UniqueName: \"kubernetes.io/projected/66623a2e-1062-4b84-99fe-7af3b038c072-kube-api-access-mvg6g\") pod \"simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") " pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:37.833245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.832239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:37.864011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.863973    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:37.864175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.864028    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 19:19:37.885536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:37.885507    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:38.065722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:38.065568    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:38.065722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:38.065610    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:19:38.225713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:38.212753    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"]
Oct 30 19:19:38.246036 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:19:38.245995    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66623a2e_1062_4b84_99fe_7af3b038c072.slice/crio-765c09c5aa986fb4c260da122126dae8a37f72b4ffe2d298946e11808284d91e WatchSource:0}: Error finding container 765c09c5aa986fb4c260da122126dae8a37f72b4ffe2d298946e11808284d91e: Status 404 returned error can't find the container with id 765c09c5aa986fb4c260da122126dae8a37f72b4ffe2d298946e11808284d91e
Oct 30 19:19:38.562720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:38.562611    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:38.864233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:38.864149    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:19:38.864233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:38.864202    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 19:19:39.037596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:39.037558    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" event={"ID":"66623a2e-1062-4b84-99fe-7af3b038c072","Type":"ContainerStarted","Data":"c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d"}
Oct 30 19:19:39.037916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:39.037897    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" event={"ID":"66623a2e-1062-4b84-99fe-7af3b038c072","Type":"ContainerStarted","Data":"765c09c5aa986fb4c260da122126dae8a37f72b4ffe2d298946e11808284d91e"}
Oct 30 19:19:39.056354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:39.056274    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podStartSLOduration=2.056258514 podStartE2EDuration="2.056258514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:39.054842104 +0000 UTC m=+813.957855140" watchObservedRunningTime="2026-10-30 19:19:39.056258514 +0000 UTC m=+813.959271551"
Oct 30 19:19:39.065752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:39.065723    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:19:39.065883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:39.065775    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 30 19:19:39.433481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:39.433444    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:39.868294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:39.868221    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:39.869315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:39.869294    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:19:39.886051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:39.886031    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:40.069084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:40.069056    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:40.070275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:40.070255    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:19:40.177401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:40.177365    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:41.152433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:41.152394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:41.773678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:41.773647    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:41.930294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:41.930262    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:42.489915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:42.489882    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:43.260672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:43.260633    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:43.573072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:43.572991    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:43.586297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:43.586273    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:43.853633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:43.853538    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:44.077237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.077208    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:19:44.462332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.462289    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:44.597615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.593959    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"]
Oct 30 19:19:44.600000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.599971    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.606017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.605997    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2d5d53\"/\"openshift-service-ca.crt\""
Oct 30 19:19:44.606896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.606875    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2d5d53\"/\"default-dockercfg-8rj6n\""
Oct 30 19:19:44.607014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.606894    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2d5d53\"/\"kube-root-ca.crt\""
Oct 30 19:19:44.607014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.606952    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2d5d53\"/\"simulator-w5-2d5d53-kserve-self-signed-certs\""
Oct 30 19:19:44.607968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.607953    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"]
Oct 30 19:19:44.693948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.693910    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234d689f-2905-4f14-b56d-2906a7d4667c-tls-certs\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.694142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.693965    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-dshm\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.694142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.694082    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-tmp-dir\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.694142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.694117    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-model-cache\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.694323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.694144    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-home\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.694323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.694261    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcc4j\" (UniqueName: \"kubernetes.io/projected/234d689f-2905-4f14-b56d-2906a7d4667c-kube-api-access-jcc4j\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.698405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.698374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:44.795345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.795262    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234d689f-2905-4f14-b56d-2906a7d4667c-tls-certs\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.795345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.795328    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-dshm\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.795578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.795371    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-tmp-dir\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.795578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.795410    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-model-cache\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.795578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.795445    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-home\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.795578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.795514    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcc4j\" (UniqueName: \"kubernetes.io/projected/234d689f-2905-4f14-b56d-2906a7d4667c-kube-api-access-jcc4j\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.795951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.795908    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-tmp-dir\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.796084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.795968    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-home\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.796084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.795978    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-model-cache\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.797961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.797934    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-dshm\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.798138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.798122    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234d689f-2905-4f14-b56d-2906a7d4667c-tls-certs\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.807197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.807168    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcc4j\" (UniqueName: \"kubernetes.io/projected/234d689f-2905-4f14-b56d-2906a7d4667c-kube-api-access-jcc4j\") pod \"simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") " pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:44.914772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:44.914728    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:45.181048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:45.181010    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:45.191905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:45.191884    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"]
Oct 30 19:19:45.194860 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:19:45.194833    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod234d689f_2905_4f14_b56d_2906a7d4667c.slice/crio-5fe0221d07941ec45e8c364442633661b6d0840d2a3574dc0d4ec7b6ee17dd4d WatchSource:0}: Error finding container 5fe0221d07941ec45e8c364442633661b6d0840d2a3574dc0d4ec7b6ee17dd4d: Status 404 returned error can't find the container with id 5fe0221d07941ec45e8c364442633661b6d0840d2a3574dc0d4ec7b6ee17dd4d
Oct 30 19:19:45.941745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:45.941711    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:46.077070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:46.077039    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" event={"ID":"234d689f-2905-4f14-b56d-2906a7d4667c","Type":"ContainerStarted","Data":"7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7"}
Oct 30 19:19:46.077265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:46.077083    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" event={"ID":"234d689f-2905-4f14-b56d-2906a7d4667c","Type":"ContainerStarted","Data":"5fe0221d07941ec45e8c364442633661b6d0840d2a3574dc0d4ec7b6ee17dd4d"}
Oct 30 19:19:46.099790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:46.099740    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podStartSLOduration=2.099725861 podStartE2EDuration="2.099725861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:46.09827668 +0000 UTC m=+821.001289717" watchObservedRunningTime="2026-10-30 19:19:46.099725861 +0000 UTC m=+821.002738897"
Oct 30 19:19:46.601059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:46.601009    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:46.801729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:46.801670    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:46.915639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:46.915612    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:47.451229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.451197    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:47.867922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.867838    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"]
Oct 30 19:19:47.870443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.870410    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:47.875433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.875400    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"simulator-w0-ad23c7-kserve-self-signed-certs\""
Oct 30 19:19:47.875659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.875506    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"openshift-service-ca.crt\""
Oct 30 19:19:47.875659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.875641    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"kube-root-ca.crt\""
Oct 30 19:19:47.875805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.875765    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"default-dockercfg-lxwmh\""
Oct 30 19:19:47.887008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.886256    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:47.889121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.889098    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"]
Oct 30 19:19:47.905389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.905364    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:47.981376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:47.981342    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:48.025942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.025912    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-home\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.026123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.026088    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tmp-dir\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.026181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.026144    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-dshm\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.026258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.026234    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-model-cache\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.026320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.026254    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p69cn\" (UniqueName: \"kubernetes.io/projected/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-kube-api-access-p69cn\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.026320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.026295    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tls-certs\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.098280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.098254    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:19:48.128207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.128101    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tmp-dir\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.128207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.128178    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-dshm\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.128461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.128214    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-model-cache\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.128461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.128244    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p69cn\" (UniqueName: \"kubernetes.io/projected/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-kube-api-access-p69cn\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.128461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.128294    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tls-certs\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.128461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.128357    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-home\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.128950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.128925    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-home\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.129221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.129185    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tmp-dir\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.130901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.130041    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-model-cache\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.156002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.155957    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p69cn\" (UniqueName: \"kubernetes.io/projected/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-kube-api-access-p69cn\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.169736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.166217    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-dshm\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.169736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.166239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:48.169736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.168386    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tls-certs\") pod \"simulator-w0-ad23c7-kserve-576499569f-bkjmk\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") " pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.182463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.182226    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:48.468773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.468745    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"]
Oct 30 19:19:48.471040 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:19:48.471011    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44c77b77_cb30_4abf_ba5a_1747dae4d2e1.slice/crio-84cb75cb778032761dc4c00e88dce16366102fd602bb544488dddea11bc32c16 WatchSource:0}: Error finding container 84cb75cb778032761dc4c00e88dce16366102fd602bb544488dddea11bc32c16: Status 404 returned error can't find the container with id 84cb75cb778032761dc4c00e88dce16366102fd602bb544488dddea11bc32c16
Oct 30 19:19:48.761491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:48.761393    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:49.093931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:49.093839    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" event={"ID":"44c77b77-cb30-4abf-ba5a-1747dae4d2e1","Type":"ContainerStarted","Data":"be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2"}
Oct 30 19:19:49.093931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:49.093883    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" event={"ID":"44c77b77-cb30-4abf-ba5a-1747dae4d2e1","Type":"ContainerStarted","Data":"84cb75cb778032761dc4c00e88dce16366102fd602bb544488dddea11bc32c16"}
Oct 30 19:19:49.121293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:49.121251    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podStartSLOduration=2.121237517 podStartE2EDuration="2.121237517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:49.11910278 +0000 UTC m=+824.022115818" watchObservedRunningTime="2026-10-30 19:19:49.121237517 +0000 UTC m=+824.024250553"
Oct 30 19:19:49.182946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:49.182919    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:49.438445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:49.438413    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:50.488589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:50.488551    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:51.289587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:51.289532    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:51.956976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:51.956939    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:52.128018 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:19:52.127982    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:19:52.367874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:52.367781    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:52.762333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:52.762276    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:53.580455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:53.580419    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:54.375248 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:54.375196    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:54.915754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:54.915715    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:54.928505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:54.928481    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:55.143027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:55.143002    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:19:55.180417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:55.180320    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:55.624066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:55.623958    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:56.124439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:56.124410    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:56.884424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:56.884388    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:57.507085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:57.507050    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:58.178908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:58.178873    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:58.182725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:58.182679    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:58.195374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:58.195353    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:58.836538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:58.836506    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:59.162670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:59.162644    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:19:59.570110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:19:59.570019    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:00.155249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:00.155201    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:00.747250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:00.747211    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:01.553272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:01.553236    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:02.226275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:02.226242    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:02.877705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:02.877658    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:03.553899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.553869    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:03.806841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.806750    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"]
Oct 30 19:20:03.811410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.811383    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.817940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.817913    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39b867\"/\"premium-w2-39b867-kserve-self-signed-certs\""
Oct 30 19:20:03.820071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.819579    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"]
Oct 30 19:20:03.894708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.894659    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tmp-dir\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.895173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.894732    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tls-certs\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.895173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.894843    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-dshm\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.895173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.894890    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-home\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.895173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.895018    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-model-cache\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.895173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.895113    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqjl\" (UniqueName: \"kubernetes.io/projected/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-kube-api-access-2tqjl\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.915317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.915277    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:03.995749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.995716    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-model-cache\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.995942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.995788    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqjl\" (UniqueName: \"kubernetes.io/projected/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-kube-api-access-2tqjl\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.995942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.995858    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tmp-dir\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.995942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.995897    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tls-certs\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.995942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.995930    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-dshm\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.996155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.995957    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-home\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.996208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.996183    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-model-cache\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.996285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.996263    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tmp-dir\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.996378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.996360    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-home\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.998435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.998407    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-dshm\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:03.998673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:03.998652    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tls-certs\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:04.008217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:04.008182    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqjl\" (UniqueName: \"kubernetes.io/projected/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-kube-api-access-2tqjl\") pod \"premium-w2-39b867-kserve-7ff487f55f-s9lgg\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") " pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:04.138940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:04.138839    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:04.241105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:04.241074    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:04.421009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:04.420864    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"]
Oct 30 19:20:04.429275 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:04.429243    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1ab7137_f18d_4dca_ae03_34a20e2fda7f.slice/crio-34e549ba9bebd2cb556109f22d97bfc7a86fe0ce3ac3c0c18fa70e871807fc63 WatchSource:0}: Error finding container 34e549ba9bebd2cb556109f22d97bfc7a86fe0ce3ac3c0c18fa70e871807fc63: Status 404 returned error can't find the container with id 34e549ba9bebd2cb556109f22d97bfc7a86fe0ce3ac3c0c18fa70e871807fc63
Oct 30 19:20:04.964617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:04.964578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:05.194940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:05.194895    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" event={"ID":"e1ab7137-f18d-4dca-ae03-34a20e2fda7f","Type":"ContainerStarted","Data":"d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72"}
Oct 30 19:20:05.195110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:05.194950    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" event={"ID":"e1ab7137-f18d-4dca-ae03-34a20e2fda7f","Type":"ContainerStarted","Data":"34e549ba9bebd2cb556109f22d97bfc7a86fe0ce3ac3c0c18fa70e871807fc63"}
Oct 30 19:20:05.214675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:05.214561    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podStartSLOduration=2.214543979 podStartE2EDuration="2.214543979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:05.213609335 +0000 UTC m=+840.116622372" watchObservedRunningTime="2026-10-30 19:20:05.214543979 +0000 UTC m=+840.117557017"
Oct 30 19:20:05.577330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:05.577241    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:06.139703 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:06.139655    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:06.269219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:06.269183    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:06.979066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:06.979031    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:07.146816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:07.146782    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:07.572514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:07.572477    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:08.509594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:08.509556    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:09.249137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:09.249105    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:09.852086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:09.852046    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:10.203303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:10.203274    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:10.600587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:10.600490    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:11.553917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:11.553889    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:12.000163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.000132    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"]
Oct 30 19:20:12.006049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.006025    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.010877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.010855    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bce71f\"/\"kube-root-ca.crt\""
Oct 30 19:20:12.013375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.013350    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"]
Oct 30 19:20:12.013619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.013603    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bce71f\"/\"default-dockercfg-rgxzp\""
Oct 30 19:20:12.014371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.014351    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bce71f\"/\"premium-w6-bce71f-kserve-self-signed-certs\""
Oct 30 19:20:12.017555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.016317    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bce71f\"/\"openshift-service-ca.crt\""
Oct 30 19:20:12.081089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.081060    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-model-cache\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.081272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.081206    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-dshm\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.081272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.081237    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/6951302f-5e56-4c26-ac90-10b26f64724a-kube-api-access-vwbv2\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.081373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.081278    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-tmp-dir\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.081458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.081437    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6951302f-5e56-4c26-ac90-10b26f64724a-tls-certs\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.081508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.081484    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-home\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.116975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.116946    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:12.182851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.182808    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-home\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.183055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.182875    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-model-cache\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.183055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.182973    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-dshm\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.183055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.183002    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/6951302f-5e56-4c26-ac90-10b26f64724a-kube-api-access-vwbv2\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.183055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.183045    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-tmp-dir\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.183290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.183109    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6951302f-5e56-4c26-ac90-10b26f64724a-tls-certs\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.183345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.183295    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-home\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.183402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.183346    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-model-cache\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.183838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.183813    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-tmp-dir\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.186266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.186241    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6951302f-5e56-4c26-ac90-10b26f64724a-tls-certs\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.188030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.187529    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-dshm\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.196590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.196562    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/6951302f-5e56-4c26-ac90-10b26f64724a-kube-api-access-vwbv2\") pod \"premium-w6-bce71f-kserve-74556658b4-g8p7s\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") " pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.352039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.351929    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:12.385037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.385001    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:12.643828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:12.643799    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"]
Oct 30 19:20:12.647185 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:12.647157    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6951302f_5e56_4c26_ac90_10b26f64724a.slice/crio-978e6452e9e9d1cefbba7003f2276e0dc67f1d881e8e872662ae2df7f2bb7c7b WatchSource:0}: Error finding container 978e6452e9e9d1cefbba7003f2276e0dc67f1d881e8e872662ae2df7f2bb7c7b: Status 404 returned error can't find the container with id 978e6452e9e9d1cefbba7003f2276e0dc67f1d881e8e872662ae2df7f2bb7c7b
Oct 30 19:20:13.165054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:13.165013    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:13.260745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:13.260713    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" event={"ID":"6951302f-5e56-4c26-ac90-10b26f64724a","Type":"ContainerStarted","Data":"a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b"}
Oct 30 19:20:13.260908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:13.260752    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" event={"ID":"6951302f-5e56-4c26-ac90-10b26f64724a","Type":"ContainerStarted","Data":"978e6452e9e9d1cefbba7003f2276e0dc67f1d881e8e872662ae2df7f2bb7c7b"}
Oct 30 19:20:13.291965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:13.291919    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podStartSLOduration=2.291905206 podStartE2EDuration="2.291905206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:13.290832754 +0000 UTC m=+848.193845792" watchObservedRunningTime="2026-10-30 19:20:13.291905206 +0000 UTC m=+848.194918242"
Oct 30 19:20:13.352884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:13.352856    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:14.015045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:14.015011    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:14.139119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:14.139065    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:14.152121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:14.152100    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:14.274094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:14.274022    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:20:14.747096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:14.747058    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:15.470159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:15.470124    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:16.075229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:16.075155    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:16.752296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:16.752257    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:17.582261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:17.582222    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:18.182396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:18.182360    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:19.046432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.046399    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:19.450514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.450482    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"]
Oct 30 19:20:19.455228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.455208    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.459216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.459192    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cfa4bb-openshift-default-dockercfg-jqxm7\""
Oct 30 19:20:19.466296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.466272    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"]
Oct 30 19:20:19.561837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.561805    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.562006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.561877    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.562051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.562011    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.562095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.562072    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.562132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.562121    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.562190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.562153    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m52lq\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-kube-api-access-m52lq\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.562264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.562248    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.562331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.562284    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.562389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.562331    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.663461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.663426    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.663666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.663472    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.663666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.663521    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.663666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.663577    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.663666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.663636    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.663666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.663662    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.663950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.663708    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m52lq\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-kube-api-access-m52lq\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.663950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.663747    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.663950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.663770    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.664611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.664587    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.664817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.664794    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.665149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.665123    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.665254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.665134    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.665445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.665426    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.666593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.666576    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.667242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.667224    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.684021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.683999    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.684479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.684463    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m52lq\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-kube-api-access-m52lq\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:19.697011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.696985    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:19.770166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:19.770094    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:20.101807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:20.101721    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"]
Oct 30 19:20:20.106477 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:20.106447    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36e9eb9_021f_4c57_ab8a_d13cc9bc5c02.slice/crio-f05f2203b36132b290afc812ee4c60c9fd4af347ab8e247343c44eb645e90b47 WatchSource:0}: Error finding container f05f2203b36132b290afc812ee4c60c9fd4af347ab8e247343c44eb645e90b47: Status 404 returned error can't find the container with id f05f2203b36132b290afc812ee4c60c9fd4af347ab8e247343c44eb645e90b47
Oct 30 19:20:20.108980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:20.108950    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:20:20.109036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:20.109014    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:20:20.109077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:20.109041    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:20:20.296857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:20.296816    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" event={"ID":"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02","Type":"ContainerStarted","Data":"2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740"}
Oct 30 19:20:20.297029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:20.296885    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" event={"ID":"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02","Type":"ContainerStarted","Data":"f05f2203b36132b290afc812ee4c60c9fd4af347ab8e247343c44eb645e90b47"}
Oct 30 19:20:20.327914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:20.327858    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" podStartSLOduration=1.327842334 podStartE2EDuration="1.327842334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:20.326392825 +0000 UTC m=+855.229405863" watchObservedRunningTime="2026-10-30 19:20:20.327842334 +0000 UTC m=+855.230855371"
Oct 30 19:20:20.671894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:20.671862    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:20.773824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:20.773793    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:21.291458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.291423    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:21.772474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.772436    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:20:21.772887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.772496    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:20:21.884355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.884322    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"]
Oct 30 19:20:21.888537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.888516    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:21.892582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.892558    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2d5d53\"/\"premium-w5-2d5d53-kserve-self-signed-certs\""
Oct 30 19:20:21.928786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.928759    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"]
Oct 30 19:20:21.934547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.934514    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:21.991928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.991894    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9cr\" (UniqueName: \"kubernetes.io/projected/0a520025-69b4-4288-b951-0333b96ed430-kube-api-access-tq9cr\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:21.992102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.991935    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-dshm\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:21.992102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.991961    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a520025-69b4-4288-b951-0333b96ed430-tls-certs\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:21.992221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.992165    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-model-cache\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:21.992294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.992266    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-tmp-dir\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:21.992338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:21.992323    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-home\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.094120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.094027    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9cr\" (UniqueName: \"kubernetes.io/projected/0a520025-69b4-4288-b951-0333b96ed430-kube-api-access-tq9cr\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.094120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.094089    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-dshm\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.094294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.094124    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a520025-69b4-4288-b951-0333b96ed430-tls-certs\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.094294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.094206    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-model-cache\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.094294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.094270    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-tmp-dir\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.094397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.094313    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-home\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.094874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.094847    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-home\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.095085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.094904    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-tmp-dir\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.098709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.095777    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-model-cache\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.108794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.108771    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:22.112422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.112399    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9cr\" (UniqueName: \"kubernetes.io/projected/0a520025-69b4-4288-b951-0333b96ed430-kube-api-access-tq9cr\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.139376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.139348    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a520025-69b4-4288-b951-0333b96ed430-tls-certs\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.139753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.139734    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-dshm\") pod \"premium-w5-2d5d53-kserve-5549c5ff96-488bk\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") " pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.164610 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:20:22.164581    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:20:22.236384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.236352    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:22.352608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.352528    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:22.365657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.365634    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:22.531559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.531537    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"]
Oct 30 19:20:22.533671 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:22.533649    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a520025_69b4_4288_b951_0333b96ed430.slice/crio-8f294be4160b2e73894f1c491c91e208ecd6bf25626cabe9b456bd25476f86bb WatchSource:0}: Error finding container 8f294be4160b2e73894f1c491c91e208ecd6bf25626cabe9b456bd25476f86bb: Status 404 returned error can't find the container with id 8f294be4160b2e73894f1c491c91e208ecd6bf25626cabe9b456bd25476f86bb
Oct 30 19:20:22.679178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.679146    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:22.771375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.771342    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:20:22.771541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:22.771387    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:20:23.338856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:23.338821    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" event={"ID":"0a520025-69b4-4288-b951-0333b96ed430","Type":"ContainerStarted","Data":"345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853"}
Oct 30 19:20:23.338856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:23.338864    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" event={"ID":"0a520025-69b4-4288-b951-0333b96ed430","Type":"ContainerStarted","Data":"8f294be4160b2e73894f1c491c91e208ecd6bf25626cabe9b456bd25476f86bb"}
Oct 30 19:20:23.342275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:23.342248    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:23.350795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:23.350773    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:20:23.358881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:23.358841    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podStartSLOduration=2.358829198 podStartE2EDuration="2.358829198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:23.356449805 +0000 UTC m=+858.259462841" watchObservedRunningTime="2026-10-30 19:20:23.358829198 +0000 UTC m=+858.261842235"
Oct 30 19:20:23.770593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:23.770562    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:20:23.770801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:23.770614    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:20:24.062983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:24.062894    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:24.236965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:24.236928    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:24.683713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:24.683648    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:24.771235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:24.771199    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:20:24.771392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:24.771251    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:20:25.406270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:25.406238    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:25.770642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:25.770553    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:20:25.770642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:25.770614    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:20:26.193989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:26.193959    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:26.483789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:26.483702    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:26.901904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:26.901872    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:26.902358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:26.902299    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:26.903205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:26.903184    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:20:27.104631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:27.104588    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:27.765766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:27.765707    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:28.595624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:28.595593    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:29.437314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:29.437283    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:30.245388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:30.245352    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:30.885557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:30.885522    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:31.493643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:31.493611    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:32.236950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:32.236915    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:32.251222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:32.251199    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:32.258971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:32.258941    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:32.433802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:32.433769    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:20:33.047811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:33.047777    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:34.066983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.066947    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:34.547748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.547718    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"]
Oct 30 19:20:34.551511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.551496    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.556829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.556808    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-27b5d5-openshift-default-dockercfg-djg8x\""
Oct 30 19:20:34.566016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.565988    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"]
Oct 30 19:20:34.646388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.646356    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrjs\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-kube-api-access-fdrjs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.646558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.646470    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-credential-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.646558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.646508    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-certs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.646638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.646582    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.646638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.646611    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-data\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.646638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.646630    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istiod-ca-cert\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.646789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.646770    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-envoy\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.646865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.646850    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-token\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.646899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.646875    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-podinfo\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748034    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-credential-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748071    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-certs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748133    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748161    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-data\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748275    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istiod-ca-cert\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748374    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-envoy\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748461    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-credential-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748470    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-token\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748487    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-data\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748522    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-podinfo\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748562    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-certs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748591    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-socket\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.748726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.748645    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrjs\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-kube-api-access-fdrjs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.749062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.749009    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istiod-ca-cert\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.750680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.750662    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-envoy\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.751004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.750986    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-podinfo\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.760580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.760544    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-token\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.760857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.760819    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrjs\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-kube-api-access-fdrjs\") pod \"e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") " pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:34.864102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.864009    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:34.865918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:34.865896    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:35.154698 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:35.154660    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"]
Oct 30 19:20:35.157310 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:35.157286    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeb17ec_a2bf_4a76_8d21_7edc62e2c915.slice/crio-f8369943fd75e2ded6ee87a45376adf83db36879cce9b1bd802d32e02ed3d5a8 WatchSource:0}: Error finding container f8369943fd75e2ded6ee87a45376adf83db36879cce9b1bd802d32e02ed3d5a8: Status 404 returned error can't find the container with id f8369943fd75e2ded6ee87a45376adf83db36879cce9b1bd802d32e02ed3d5a8
Oct 30 19:20:35.159985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:35.159941    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:20:35.160042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:35.160026    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:20:35.160075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:35.160053    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:20:35.436188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:35.436090    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" event={"ID":"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915","Type":"ContainerStarted","Data":"829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3"}
Oct 30 19:20:35.436188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:35.436160    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" event={"ID":"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915","Type":"ContainerStarted","Data":"f8369943fd75e2ded6ee87a45376adf83db36879cce9b1bd802d32e02ed3d5a8"}
Oct 30 19:20:35.458725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:35.458656    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" podStartSLOduration=1.458641426 podStartE2EDuration="1.458641426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:35.457315754 +0000 UTC m=+870.360328793" watchObservedRunningTime="2026-10-30 19:20:35.458641426 +0000 UTC m=+870.361654465"
Oct 30 19:20:35.650152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:35.650091    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:35.866262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:35.866179    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:36.263763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:36.263731    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:36.871613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:36.871580    4925 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:36.871975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:36.871943    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:20:36.887321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:36.887290    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:37.740713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:37.740652    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:37.866223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:37.866193    4925 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:37.866396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:37.866248    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:20:38.866536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:38.866503    4925 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:38.867063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:38.866561    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:20:39.253201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:39.253166    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:39.689713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:39.686875    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:39.874416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:39.874379    4925 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:39.875010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:39.874981    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:20:40.554638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:40.554605    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:40.867013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:40.866935    4925 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:40.867013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:40.866984    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:20:41.211713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:41.210212    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:41.867207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:41.867173    4925 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:20:41.867383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:41.867225    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:20:41.990982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:41.990946    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:42.651004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:42.650973    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:43.155219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:43.155148    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:43.155682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:43.155656    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:43.156777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:43.156757    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:20:43.379226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:43.379196    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:44.044019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.043983    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:44.704878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.704851    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"]
Oct 30 19:20:44.708591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.708573    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:44.715478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.715459    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:44.715820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.715800    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:20:44.719043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.719020    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"]
Oct 30 19:20:44.772746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.772715    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7962588-dffa-473a-a502-6c26ee508933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4aea515-vkl82\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:44.772898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.772755    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pf9\" (UniqueName: \"kubernetes.io/projected/b7962588-dffa-473a-a502-6c26ee508933-kube-api-access-85pf9\") pod \"maas-api-revoke-sub-e4aea515-vkl82\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:44.772986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.772966    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7962588-dffa-473a-a502-6c26ee508933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4aea515-vkl82\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:44.857788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.857748    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:44.873956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.873926    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7962588-dffa-473a-a502-6c26ee508933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4aea515-vkl82\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:44.874118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.873999    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7962588-dffa-473a-a502-6c26ee508933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4aea515-vkl82\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:44.874118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.874033    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pf9\" (UniqueName: \"kubernetes.io/projected/b7962588-dffa-473a-a502-6c26ee508933-kube-api-access-85pf9\") pod \"maas-api-revoke-sub-e4aea515-vkl82\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:44.874564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.874545    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7962588-dffa-473a-a502-6c26ee508933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4aea515-vkl82\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:44.876493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.876469    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7962588-dffa-473a-a502-6c26ee508933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4aea515-vkl82\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:44.889042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:44.889021    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pf9\" (UniqueName: \"kubernetes.io/projected/b7962588-dffa-473a-a502-6c26ee508933-kube-api-access-85pf9\") pod \"maas-api-revoke-sub-e4aea515-vkl82\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:45.024361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:45.024290    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:45.309110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:45.309089    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"]
Oct 30 19:20:45.311897 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:45.311870    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7962588_dffa_473a_a502_6c26ee508933.slice/crio-34548e30da448c6f9ff298dc12c6099fb9ca6bf8fec7687bc2f30ec2698be709 WatchSource:0}: Error finding container 34548e30da448c6f9ff298dc12c6099fb9ca6bf8fec7687bc2f30ec2698be709: Status 404 returned error can't find the container with id 34548e30da448c6f9ff298dc12c6099fb9ca6bf8fec7687bc2f30ec2698be709
Oct 30 19:20:45.491550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:45.491518    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82" event={"ID":"b7962588-dffa-473a-a502-6c26ee508933","Type":"ContainerStarted","Data":"34548e30da448c6f9ff298dc12c6099fb9ca6bf8fec7687bc2f30ec2698be709"}
Oct 30 19:20:45.651876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:45.651842    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:46.467285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.467248    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:46.814323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.814237    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"]
Oct 30 19:20:46.818167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.818149    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:46.826501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.826480    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"]
Oct 30 19:20:46.894017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.893986    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6cde10f-jngdj\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:46.894171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.894027    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6cde10f-jngdj\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:46.894252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.894180    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlx8\" (UniqueName: \"kubernetes.io/projected/2c0a53ab-a863-4c36-b396-1ae726360bef-kube-api-access-wqlx8\") pod \"maas-api-revoke-sub-d6cde10f-jngdj\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:46.995175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.995130    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlx8\" (UniqueName: \"kubernetes.io/projected/2c0a53ab-a863-4c36-b396-1ae726360bef-kube-api-access-wqlx8\") pod \"maas-api-revoke-sub-d6cde10f-jngdj\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:46.995329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.995261    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6cde10f-jngdj\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:46.995329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.995286    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6cde10f-jngdj\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:46.996078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.996048    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6cde10f-jngdj\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:46.998162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:46.998136    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6cde10f-jngdj\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:47.007251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:47.007229    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlx8\" (UniqueName: \"kubernetes.io/projected/2c0a53ab-a863-4c36-b396-1ae726360bef-kube-api-access-wqlx8\") pod \"maas-api-revoke-sub-d6cde10f-jngdj\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:47.107908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:47.107813    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:47.144775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:47.144744    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:47.434219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:47.434198    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"]
Oct 30 19:20:47.436451 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:47.436424    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0a53ab_a863_4c36_b396_1ae726360bef.slice/crio-03ce1eac7b3a410b19252ca4490e62d6c2705d4ed9735d5a871dc586ed2201c2 WatchSource:0}: Error finding container 03ce1eac7b3a410b19252ca4490e62d6c2705d4ed9735d5a871dc586ed2201c2: Status 404 returned error can't find the container with id 03ce1eac7b3a410b19252ca4490e62d6c2705d4ed9735d5a871dc586ed2201c2
Oct 30 19:20:47.504201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:47.504172    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj" event={"ID":"2c0a53ab-a863-4c36-b396-1ae726360bef","Type":"ContainerStarted","Data":"03ce1eac7b3a410b19252ca4490e62d6c2705d4ed9735d5a871dc586ed2201c2"}
Oct 30 19:20:47.756890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:47.756133    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:48.480326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:48.480284    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:49.115914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.115810    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:49.527732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.527700    4925 generic.go:356] "Generic (PLEG): container finished" podID="b7962588-dffa-473a-a502-6c26ee508933" containerID="5bb3527282f0c55def48bed3b1ff1381f251e68e06eaf49acbbc7bb869dbb6a6" exitCode=0
Oct 30 19:20:49.527928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.527781    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82" event={"ID":"b7962588-dffa-473a-a502-6c26ee508933","Type":"ContainerDied","Data":"5bb3527282f0c55def48bed3b1ff1381f251e68e06eaf49acbbc7bb869dbb6a6"}
Oct 30 19:20:49.529182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.529156    4925 generic.go:356] "Generic (PLEG): container finished" podID="2c0a53ab-a863-4c36-b396-1ae726360bef" containerID="dd9108a45c469736a07ed524e20933e2b0fd5bf578a890c19f43779a1a124eec" exitCode=0
Oct 30 19:20:49.529281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.529247    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj" event={"ID":"2c0a53ab-a863-4c36-b396-1ae726360bef","Type":"ContainerDied","Data":"dd9108a45c469736a07ed524e20933e2b0fd5bf578a890c19f43779a1a124eec"}
Oct 30 19:20:49.754311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.754269    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:49.840439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.840358    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"]
Oct 30 19:20:49.844207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.844188    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:49.847230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.847198    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-17e904-openshift-default-dockercfg-d5z8k\""
Oct 30 19:20:49.857754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.857729    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"]
Oct 30 19:20:49.927381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.927350    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:49.927381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.927385    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6flhl\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-kube-api-access-6flhl\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:49.927647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.927432    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:49.927647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.927543    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:49.927647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.927604    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a3e104-73aa-4209-a725-c333f0b7ecd8-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:49.927807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.927666    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:49.927807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.927739    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:49.927807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.927770    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:49.927807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:49.927794    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.028752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.028709    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.028752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.028759    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6flhl\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-kube-api-access-6flhl\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.028819    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.028852    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.028906    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a3e104-73aa-4209-a725-c333f0b7ecd8-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.028935    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.028971    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.029007    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.029068    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.029282    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.029357    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.029398    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.029599    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.029722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.029670    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a3e104-73aa-4209-a725-c333f0b7ecd8-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.031373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.031347    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.031621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.031577    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.041040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.041017    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.041663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.041642    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flhl\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-kube-api-access-6flhl\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.158064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.158024    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:50.450836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.450723    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"]
Oct 30 19:20:50.453644 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:50.453613    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a3e104_73aa_4209_a725_c333f0b7ecd8.slice/crio-fdb9aaaeb80f30106ec9b2de01909e1922e36d21385169f05595cdb065a6fb6a WatchSource:0}: Error finding container fdb9aaaeb80f30106ec9b2de01909e1922e36d21385169f05595cdb065a6fb6a: Status 404 returned error can't find the container with id fdb9aaaeb80f30106ec9b2de01909e1922e36d21385169f05595cdb065a6fb6a
Oct 30 19:20:50.456093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.456061    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:20:50.456163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.456123    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:20:50.456163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.456149    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:20:50.537617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.537583    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" event={"ID":"66a3e104-73aa-4209-a725-c333f0b7ecd8","Type":"ContainerStarted","Data":"e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0"}
Oct 30 19:20:50.537780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.537643    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" event={"ID":"66a3e104-73aa-4209-a725-c333f0b7ecd8","Type":"ContainerStarted","Data":"fdb9aaaeb80f30106ec9b2de01909e1922e36d21385169f05595cdb065a6fb6a"}
Oct 30 19:20:50.567101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.567039    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" podStartSLOduration=1.567019868 podStartE2EDuration="1.567019868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:50.562799549 +0000 UTC m=+885.465812586" watchObservedRunningTime="2026-10-30 19:20:50.567019868 +0000 UTC m=+885.470032906"
Oct 30 19:20:50.674742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.674628    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:50.798071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.798044    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:50.803824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.801921    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:50.841619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.840595    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7962588-dffa-473a-a502-6c26ee508933-kube-api-access-85pf9\" (UniqueName: \"kubernetes.io/projected/b7962588-dffa-473a-a502-6c26ee508933-kube-api-access-85pf9\") pod \"b7962588-dffa-473a-a502-6c26ee508933\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") "
Oct 30 19:20:50.841619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.840663    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-xks-ca\") pod \"2c0a53ab-a863-4c36-b396-1ae726360bef\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") "
Oct 30 19:20:50.841619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.840737    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7962588-dffa-473a-a502-6c26ee508933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7962588-dffa-473a-a502-6c26ee508933-maas-api-xks-ca\") pod \"b7962588-dffa-473a-a502-6c26ee508933\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") "
Oct 30 19:20:50.841619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.840909    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7962588-dffa-473a-a502-6c26ee508933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7962588-dffa-473a-a502-6c26ee508933-maas-api-openshift-ca\") pod \"b7962588-dffa-473a-a502-6c26ee508933\" (UID: \"b7962588-dffa-473a-a502-6c26ee508933\") "
Oct 30 19:20:50.841619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.840967    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0a53ab-a863-4c36-b396-1ae726360bef-kube-api-access-wqlx8\" (UniqueName: \"kubernetes.io/projected/2c0a53ab-a863-4c36-b396-1ae726360bef-kube-api-access-wqlx8\") pod \"2c0a53ab-a863-4c36-b396-1ae726360bef\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") "
Oct 30 19:20:50.841619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.841005    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-openshift-ca\") pod \"2c0a53ab-a863-4c36-b396-1ae726360bef\" (UID: \"2c0a53ab-a863-4c36-b396-1ae726360bef\") "
Oct 30 19:20:50.842582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.842105    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7962588-dffa-473a-a502-6c26ee508933-maas-api-openshift-ca" pod "b7962588-dffa-473a-a502-6c26ee508933" (UID: "b7962588-dffa-473a-a502-6c26ee508933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:50.842582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.842273    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-openshift-ca" pod "2c0a53ab-a863-4c36-b396-1ae726360bef" (UID: "2c0a53ab-a863-4c36-b396-1ae726360bef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:50.922713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.921825    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-xks-ca" pod "2c0a53ab-a863-4c36-b396-1ae726360bef" (UID: "2c0a53ab-a863-4c36-b396-1ae726360bef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:50.922713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.922387    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0a53ab-a863-4c36-b396-1ae726360bef-kube-api-access-wqlx8" pod "2c0a53ab-a863-4c36-b396-1ae726360bef" (UID: "2c0a53ab-a863-4c36-b396-1ae726360bef"). InnerVolumeSpecName "kube-api-access-wqlx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:50.942831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.927878    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7962588-dffa-473a-a502-6c26ee508933-maas-api-xks-ca" pod "b7962588-dffa-473a-a502-6c26ee508933" (UID: "b7962588-dffa-473a-a502-6c26ee508933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:50.942831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.931172    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7962588-dffa-473a-a502-6c26ee508933-kube-api-access-85pf9" pod "b7962588-dffa-473a-a502-6c26ee508933" (UID: "b7962588-dffa-473a-a502-6c26ee508933"). InnerVolumeSpecName "kube-api-access-85pf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:50.942831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.942299    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7962588-dffa-473a-a502-6c26ee508933-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:50.942831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.942320    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqlx8\" (UniqueName: \"kubernetes.io/projected/2c0a53ab-a863-4c36-b396-1ae726360bef-kube-api-access-wqlx8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:50.942831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.942330    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:50.942831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.942339    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85pf9\" (UniqueName: \"kubernetes.io/projected/b7962588-dffa-473a-a502-6c26ee508933-kube-api-access-85pf9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:50.942831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.942349    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0a53ab-a863-4c36-b396-1ae726360bef-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:50.942831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:50.942358    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7962588-dffa-473a-a502-6c26ee508933-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:51.162715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.161707    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:51.166713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.163564    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:51.166713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.163618    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:20:51.490390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.490348    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:51.563843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.563817    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"
Oct 30 19:20:51.564762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.564735    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82" event={"ID":"b7962588-dffa-473a-a502-6c26ee508933","Type":"ContainerDied","Data":"34548e30da448c6f9ff298dc12c6099fb9ca6bf8fec7687bc2f30ec2698be709"}
Oct 30 19:20:51.564949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.564933    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34548e30da448c6f9ff298dc12c6099fb9ca6bf8fec7687bc2f30ec2698be709"
Oct 30 19:20:51.568095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.568075    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"
Oct 30 19:20:51.568834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.568804    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj" event={"ID":"2c0a53ab-a863-4c36-b396-1ae726360bef","Type":"ContainerDied","Data":"03ce1eac7b3a410b19252ca4490e62d6c2705d4ed9735d5a871dc586ed2201c2"}
Oct 30 19:20:51.568968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:51.568954    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ce1eac7b3a410b19252ca4490e62d6c2705d4ed9735d5a871dc586ed2201c2"
Oct 30 19:20:52.168713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:52.166808    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:52.168713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:52.166870    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:20:52.251061 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:20:52.251006    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:20:52.426335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:52.426239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:53.146443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:53.146403    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:53.158871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:53.158845    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:53.159046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:53.158896    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:20:53.750798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:53.750760    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:54.159146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:54.159120    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:54.159300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:54.159167    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:20:54.657289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:54.657240    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:55.159143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:55.159111    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:55.159524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:55.159164    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:20:55.538383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:55.538279    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:55.721214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:55.721176    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:56.158933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:56.158898    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:56.159156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:56.158957    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:20:56.474754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:56.474642    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:57.158586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:57.158555    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:20:57.158787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:57.158604    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 30 19:20:57.259069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:57.259028    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:58.037932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.037892    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"]
Oct 30 19:20:58.039331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.038595    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b7962588-dffa-473a-a502-6c26ee508933" containerName="revoke-keys"
Oct 30 19:20:58.039331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.038618    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7962588-dffa-473a-a502-6c26ee508933" containerName="revoke-keys"
Oct 30 19:20:58.039331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.038664    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2c0a53ab-a863-4c36-b396-1ae726360bef" containerName="revoke-keys"
Oct 30 19:20:58.039331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.038672    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0a53ab-a863-4c36-b396-1ae726360bef" containerName="revoke-keys"
Oct 30 19:20:58.039331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.038812    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0a53ab-a863-4c36-b396-1ae726360bef" containerName="revoke-keys"
Oct 30 19:20:58.039331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.038842    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7962588-dffa-473a-a502-6c26ee508933" containerName="revoke-keys"
Oct 30 19:20:58.045656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.045623    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.051825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.049547    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"]
Oct 30 19:20:58.054436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.054415    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:58.054748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.054724    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:20:58.082757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.082737    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:58.163513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.163486    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:58.163793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.163771    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:58.164559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.164537    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:20:58.168780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.168759    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m4qf\" (UniqueName: \"kubernetes.io/projected/dfc69199-346e-4c38-8c3a-2c7162b968e3-kube-api-access-6m4qf\") pod \"maas-api-revoke-sub-1d620f5f-96f7b\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.168871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.168818    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d620f5f-96f7b\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.168920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.168888    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d620f5f-96f7b\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.269573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.269541    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d620f5f-96f7b\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.270096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.269589    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d620f5f-96f7b\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.270096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.269850    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4qf\" (UniqueName: \"kubernetes.io/projected/dfc69199-346e-4c38-8c3a-2c7162b968e3-kube-api-access-6m4qf\") pod \"maas-api-revoke-sub-1d620f5f-96f7b\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.270702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.270664    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d620f5f-96f7b\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.272541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.272517    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d620f5f-96f7b\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.282059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.282034    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m4qf\" (UniqueName: \"kubernetes.io/projected/dfc69199-346e-4c38-8c3a-2c7162b968e3-kube-api-access-6m4qf\") pod \"maas-api-revoke-sub-1d620f5f-96f7b\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.365197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.365108    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:20:58.648166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.648143    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"]
Oct 30 19:20:58.650840 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:58.650817    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc69199_346e_4c38_8c3a_2c7162b968e3.slice/crio-29b4b07f8d3a6df6f4aa8f59ed9582f34e4bba0d16c8976aae272a3aecf7257f WatchSource:0}: Error finding container 29b4b07f8d3a6df6f4aa8f59ed9582f34e4bba0d16c8976aae272a3aecf7257f: Status 404 returned error can't find the container with id 29b4b07f8d3a6df6f4aa8f59ed9582f34e4bba0d16c8976aae272a3aecf7257f
Oct 30 19:20:58.847216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.847186    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:58.978366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.978329    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"]
Oct 30 19:20:58.982313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.982296    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:58.990424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:58.990401    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"]
Oct 30 19:20:59.079387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.079355    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59glb\" (UniqueName: \"kubernetes.io/projected/e84e08a9-9727-4d86-9c25-d33155ca25d4-kube-api-access-59glb\") pod \"maas-api-revoke-sub-301bd19a-rdx5k\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.079518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.079414    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-301bd19a-rdx5k\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.079569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.079529    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-301bd19a-rdx5k\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.180900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.180821    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-301bd19a-rdx5k\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.181049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.180900    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-301bd19a-rdx5k\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.181049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.181007    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59glb\" (UniqueName: \"kubernetes.io/projected/e84e08a9-9727-4d86-9c25-d33155ca25d4-kube-api-access-59glb\") pod \"maas-api-revoke-sub-301bd19a-rdx5k\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.181622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.181597    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-301bd19a-rdx5k\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.183453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.183433    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-301bd19a-rdx5k\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.192540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.192521    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59glb\" (UniqueName: \"kubernetes.io/projected/e84e08a9-9727-4d86-9c25-d33155ca25d4-kube-api-access-59glb\") pod \"maas-api-revoke-sub-301bd19a-rdx5k\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.312783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.312753    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:20:59.593444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.593415    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"]
Oct 30 19:20:59.596067 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:20:59.596044    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84e08a9_9727_4d86_9c25_d33155ca25d4.slice/crio-e927434a57b799eedd1b887a9947c34df8829bdfeded539abd853ab3b332b41d WatchSource:0}: Error finding container e927434a57b799eedd1b887a9947c34df8829bdfeded539abd853ab3b332b41d: Status 404 returned error can't find the container with id e927434a57b799eedd1b887a9947c34df8829bdfeded539abd853ab3b332b41d
Oct 30 19:20:59.639922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.639892    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k" event={"ID":"e84e08a9-9727-4d86-9c25-d33155ca25d4","Type":"ContainerStarted","Data":"e927434a57b799eedd1b887a9947c34df8829bdfeded539abd853ab3b332b41d"}
Oct 30 19:20:59.641335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.641304    4925 generic.go:356] "Generic (PLEG): container finished" podID="dfc69199-346e-4c38-8c3a-2c7162b968e3" containerID="2d1f6e787fbcd42eb5897f65a6cfdb9206eb21526dbf424fa2ed2e412f44a93c" exitCode=0
Oct 30 19:20:59.641407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.641350    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b" event={"ID":"dfc69199-346e-4c38-8c3a-2c7162b968e3","Type":"ContainerDied","Data":"2d1f6e787fbcd42eb5897f65a6cfdb9206eb21526dbf424fa2ed2e412f44a93c"}
Oct 30 19:20:59.641407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.641394    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b" event={"ID":"dfc69199-346e-4c38-8c3a-2c7162b968e3","Type":"ContainerStarted","Data":"29b4b07f8d3a6df6f4aa8f59ed9582f34e4bba0d16c8976aae272a3aecf7257f"}
Oct 30 19:20:59.758104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:20:59.758074    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:00.380980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.380893    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:00.647805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.647773    4925 generic.go:356] "Generic (PLEG): container finished" podID="e84e08a9-9727-4d86-9c25-d33155ca25d4" containerID="ff88b1c5b1cc4d63923d790c1ebf281039b4814cca80f107b6e2a431f424703c" exitCode=0
Oct 30 19:21:00.647970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.647862    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k" event={"ID":"e84e08a9-9727-4d86-9c25-d33155ca25d4","Type":"ContainerDied","Data":"ff88b1c5b1cc4d63923d790c1ebf281039b4814cca80f107b6e2a431f424703c"}
Oct 30 19:21:00.785953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.785930    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:21:00.900825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.900740    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-openshift-ca\") pod \"dfc69199-346e-4c38-8c3a-2c7162b968e3\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") "
Oct 30 19:21:00.900825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.900783    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc69199-346e-4c38-8c3a-2c7162b968e3-kube-api-access-6m4qf\" (UniqueName: \"kubernetes.io/projected/dfc69199-346e-4c38-8c3a-2c7162b968e3-kube-api-access-6m4qf\") pod \"dfc69199-346e-4c38-8c3a-2c7162b968e3\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") "
Oct 30 19:21:00.901051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.900933    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-xks-ca\") pod \"dfc69199-346e-4c38-8c3a-2c7162b968e3\" (UID: \"dfc69199-346e-4c38-8c3a-2c7162b968e3\") "
Oct 30 19:21:00.901166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.901144    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-openshift-ca" pod "dfc69199-346e-4c38-8c3a-2c7162b968e3" (UID: "dfc69199-346e-4c38-8c3a-2c7162b968e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:00.901261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.901246    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:00.903033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.903011    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-xks-ca" pod "dfc69199-346e-4c38-8c3a-2c7162b968e3" (UID: "dfc69199-346e-4c38-8c3a-2c7162b968e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:00.903124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:00.903067    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc69199-346e-4c38-8c3a-2c7162b968e3-kube-api-access-6m4qf" pod "dfc69199-346e-4c38-8c3a-2c7162b968e3" (UID: "dfc69199-346e-4c38-8c3a-2c7162b968e3"). InnerVolumeSpecName "kube-api-access-6m4qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:01.002906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.002881    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc69199-346e-4c38-8c3a-2c7162b968e3-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.002906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.002905    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m4qf\" (UniqueName: \"kubernetes.io/projected/dfc69199-346e-4c38-8c3a-2c7162b968e3-kube-api-access-6m4qf\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.252553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.252438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:01.654493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.654468    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"
Oct 30 19:21:01.654922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.654503    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b" event={"ID":"dfc69199-346e-4c38-8c3a-2c7162b968e3","Type":"ContainerDied","Data":"29b4b07f8d3a6df6f4aa8f59ed9582f34e4bba0d16c8976aae272a3aecf7257f"}
Oct 30 19:21:01.654922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.654550    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b4b07f8d3a6df6f4aa8f59ed9582f34e4bba0d16c8976aae272a3aecf7257f"
Oct 30 19:21:01.803457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.803431    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:21:01.879042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.879007    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:01.914045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.913970    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-openshift-ca\") pod \"e84e08a9-9727-4d86-9c25-d33155ca25d4\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") "
Oct 30 19:21:01.914045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.914013    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-xks-ca\") pod \"e84e08a9-9727-4d86-9c25-d33155ca25d4\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") "
Oct 30 19:21:01.914271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.914197    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84e08a9-9727-4d86-9c25-d33155ca25d4-kube-api-access-59glb\" (UniqueName: \"kubernetes.io/projected/e84e08a9-9727-4d86-9c25-d33155ca25d4-kube-api-access-59glb\") pod \"e84e08a9-9727-4d86-9c25-d33155ca25d4\" (UID: \"e84e08a9-9727-4d86-9c25-d33155ca25d4\") "
Oct 30 19:21:01.914403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.914381    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-openshift-ca" pod "e84e08a9-9727-4d86-9c25-d33155ca25d4" (UID: "e84e08a9-9727-4d86-9c25-d33155ca25d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:01.914606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.914579    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:01.916359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.916335    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-xks-ca" pod "e84e08a9-9727-4d86-9c25-d33155ca25d4" (UID: "e84e08a9-9727-4d86-9c25-d33155ca25d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:01.916432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:01.916383    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84e08a9-9727-4d86-9c25-d33155ca25d4-kube-api-access-59glb" pod "e84e08a9-9727-4d86-9c25-d33155ca25d4" (UID: "e84e08a9-9727-4d86-9c25-d33155ca25d4"). InnerVolumeSpecName "kube-api-access-59glb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:02.015784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:02.015748    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59glb\" (UniqueName: \"kubernetes.io/projected/e84e08a9-9727-4d86-9c25-d33155ca25d4-kube-api-access-59glb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.015784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:02.015777    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84e08a9-9727-4d86-9c25-d33155ca25d4-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.584841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:02.584808    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:02.680177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:02.680140    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k" event={"ID":"e84e08a9-9727-4d86-9c25-d33155ca25d4","Type":"ContainerDied","Data":"e927434a57b799eedd1b887a9947c34df8829bdfeded539abd853ab3b332b41d"}
Oct 30 19:21:02.680857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:02.680834    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e927434a57b799eedd1b887a9947c34df8829bdfeded539abd853ab3b332b41d"
Oct 30 19:21:02.680983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:02.680919    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"
Oct 30 19:21:03.354418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:03.352367    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:03.968239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:03.968204    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:04.845284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:04.845236    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:05.086544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.086513    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"]
Oct 30 19:21:05.087010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.086995    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e84e08a9-9727-4d86-9c25-d33155ca25d4" containerName="revoke-keys"
Oct 30 19:21:05.087074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.087011    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84e08a9-9727-4d86-9c25-d33155ca25d4" containerName="revoke-keys"
Oct 30 19:21:05.087074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.087023    4925 cpu_manager.go:414] "containerMap: removing container" podUID="dfc69199-346e-4c38-8c3a-2c7162b968e3" containerName="revoke-keys"
Oct 30 19:21:05.087074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.087028    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc69199-346e-4c38-8c3a-2c7162b968e3" containerName="revoke-keys"
Oct 30 19:21:05.087168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.087126    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84e08a9-9727-4d86-9c25-d33155ca25d4" containerName="revoke-keys"
Oct 30 19:21:05.087168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.087134    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc69199-346e-4c38-8c3a-2c7162b968e3" containerName="revoke-keys"
Oct 30 19:21:05.090511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.090491    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.093566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.093544    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ad23c7-openshift-default-dockercfg-b7xt7\""
Oct 30 19:21:05.102996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.102970    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"]
Oct 30 19:21:05.154086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.154047    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.154267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.154173    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.154267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.154252    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.154398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.154301    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.154398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.154318    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.154398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.154383    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.154542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.154524    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.154585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.154569    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.154633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.154620    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6rb\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-kube-api-access-8x6rb\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.255426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255395    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.255565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255438    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.255645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255576    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.255645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255629    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6rb\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-kube-api-access-8x6rb\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.255789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255678    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.255789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255754    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.255891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255828    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.255891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255842    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.255891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255867    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.256053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255894    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.256053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.255938    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.256277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.256230    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.256403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.256312    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.256711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.256667    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.258173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.258150    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.258399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.258358    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.271582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.271555    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.271822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.271802    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6rb\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-kube-api-access-8x6rb\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.335466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.335441    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:21:05.335637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.335618    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:21:05.407440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.407418    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ad23c7-openshift-default-dockercfg-b7xt7\""
Oct 30 19:21:05.414873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.414857    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:05.651978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.651942    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:05.699631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.699606    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"]
Oct 30 19:21:05.701668 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:21:05.701645    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c002067_6cf0_430e_aeaa_f7aeb26a4cbf.slice/crio-c8e43162fc78dbb60f73b8df12fe1e6328aa044726f00275e0eb87fdbdae5422 WatchSource:0}: Error finding container c8e43162fc78dbb60f73b8df12fe1e6328aa044726f00275e0eb87fdbdae5422: Status 404 returned error can't find the container with id c8e43162fc78dbb60f73b8df12fe1e6328aa044726f00275e0eb87fdbdae5422
Oct 30 19:21:05.703897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.703865    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:05.703965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.703929    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:05.703965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:05.703955    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:06.607745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:06.604818    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:06.728519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:06.728486    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" event={"ID":"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf","Type":"ContainerStarted","Data":"35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10"}
Oct 30 19:21:06.728681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:06.728541    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" event={"ID":"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf","Type":"ContainerStarted","Data":"c8e43162fc78dbb60f73b8df12fe1e6328aa044726f00275e0eb87fdbdae5422"}
Oct 30 19:21:06.752482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:06.752435    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" podStartSLOduration=1.752421734 podStartE2EDuration="1.752421734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:06.751385518 +0000 UTC m=+901.654398546" watchObservedRunningTime="2026-10-30 19:21:06.752421734 +0000 UTC m=+901.655434770"
Oct 30 19:21:07.416714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:07.415983    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:07.419824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:07.419793    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:21:07.419940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:07.419854    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 19:21:07.589925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:07.589881    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:08.419753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:08.419326    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:21:08.419753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:08.419386    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 19:21:08.491108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:08.491073    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:09.416712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:09.416111    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:21:09.416712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:09.416176    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 19:21:09.475925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:09.475891    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:10.419070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:10.419035    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:21:10.419347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:10.419318    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 19:21:10.565312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:10.565270    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:11.417148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.417111    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:21:11.418165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.418137    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 19:21:11.455468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.455444    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"]
Oct 30 19:21:11.459529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.459510    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.474215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.474195    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"]
Oct 30 19:21:11.475666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.475644    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:11.475851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.475836    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:21:11.543969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.543941    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f54d2173-2lpgb\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.544164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.544011    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbdw\" (UniqueName: \"kubernetes.io/projected/c5487022-9741-44d0-961b-3b60fe18fd78-kube-api-access-pkbdw\") pod \"maas-api-revoke-sub-f54d2173-2lpgb\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.544236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.544175    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f54d2173-2lpgb\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.591547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.591515    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:11.645386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.645326    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbdw\" (UniqueName: \"kubernetes.io/projected/c5487022-9741-44d0-961b-3b60fe18fd78-kube-api-access-pkbdw\") pod \"maas-api-revoke-sub-f54d2173-2lpgb\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.645591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.645459    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f54d2173-2lpgb\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.645655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.645632    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f54d2173-2lpgb\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.646307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.646275    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f54d2173-2lpgb\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.648147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.648118    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f54d2173-2lpgb\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.662760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.662613    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbdw\" (UniqueName: \"kubernetes.io/projected/c5487022-9741-44d0-961b-3b60fe18fd78-kube-api-access-pkbdw\") pod \"maas-api-revoke-sub-f54d2173-2lpgb\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:11.773467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:11.773379    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:12.055093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.055055    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"]
Oct 30 19:21:12.058032 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:21:12.058006    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5487022_9741_44d0_961b_3b60fe18fd78.slice/crio-6ff147e5df37f0d2355d1557ff67b12a416e05269a6d8a72c4adc34d67541517 WatchSource:0}: Error finding container 6ff147e5df37f0d2355d1557ff67b12a416e05269a6d8a72c4adc34d67541517: Status 404 returned error can't find the container with id 6ff147e5df37f0d2355d1557ff67b12a416e05269a6d8a72c4adc34d67541517
Oct 30 19:21:12.317740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.317717    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"]
Oct 30 19:21:12.323308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.321534    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.325247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.325198    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"]
Oct 30 19:21:12.333072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.333048    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"distinct-w1-cfa4bb-kserve-self-signed-certs\""
Oct 30 19:21:12.400846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.400809    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:12.450813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.450783    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:12.451875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.451849    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:12.452462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.452430    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:21:12.458506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.458484    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-tmp-dir\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.458664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.458646    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhd8l\" (UniqueName: \"kubernetes.io/projected/527bd7b1-a500-4ea2-ac56-1632216a7d74-kube-api-access-zhd8l\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.458849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.458817    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-home\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.458928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.458912    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-dshm\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.458996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.458981    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527bd7b1-a500-4ea2-ac56-1632216a7d74-tls-certs\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.459048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.459034    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-model-cache\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.560317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.560227    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-home\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.560317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.560300    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-dshm\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.560543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.560382    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527bd7b1-a500-4ea2-ac56-1632216a7d74-tls-certs\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.560543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.560406    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-model-cache\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.560543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.560530    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-tmp-dir\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.560716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.560564    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhd8l\" (UniqueName: \"kubernetes.io/projected/527bd7b1-a500-4ea2-ac56-1632216a7d74-kube-api-access-zhd8l\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.560787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.560755    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-home\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.561512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.561423    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-tmp-dir\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.565712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.561843    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-model-cache\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.565712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.563709    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-dshm\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.572949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.572918    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527bd7b1-a500-4ea2-ac56-1632216a7d74-tls-certs\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.580300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.580275    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhd8l\" (UniqueName: \"kubernetes.io/projected/527bd7b1-a500-4ea2-ac56-1632216a7d74-kube-api-access-zhd8l\") pod \"distinct-w1-cfa4bb-kserve-846f85d976-4782n\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.587811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.587782    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:12.645675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.645644    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:12.772726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.772572    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb" event={"ID":"c5487022-9741-44d0-961b-3b60fe18fd78","Type":"ContainerStarted","Data":"d9ef8f481dc431a41791ebe2a08951f105265656b8046f33571d9d28cbd5312b"}
Oct 30 19:21:12.772726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.772660    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb" event={"ID":"c5487022-9741-44d0-961b-3b60fe18fd78","Type":"ContainerStarted","Data":"6ff147e5df37f0d2355d1557ff67b12a416e05269a6d8a72c4adc34d67541517"}
Oct 30 19:21:12.793411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.793361    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb" podStartSLOduration=1.793347692 podStartE2EDuration="1.793347692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:12.792632958 +0000 UTC m=+907.695646002" watchObservedRunningTime="2026-10-30 19:21:12.793347692 +0000 UTC m=+907.696360728"
Oct 30 19:21:12.925599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:12.925575    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"]
Oct 30 19:21:12.928326 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:21:12.928293    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod527bd7b1_a500_4ea2_ac56_1632216a7d74.slice/crio-4857a4c76ac083a6522bafcee17322ac69cd84e6fbfd381f1333a75013bdd0fa WatchSource:0}: Error finding container 4857a4c76ac083a6522bafcee17322ac69cd84e6fbfd381f1333a75013bdd0fa: Status 404 returned error can't find the container with id 4857a4c76ac083a6522bafcee17322ac69cd84e6fbfd381f1333a75013bdd0fa
Oct 30 19:21:13.292034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:13.291951    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:13.779035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:13.779001    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" event={"ID":"527bd7b1-a500-4ea2-ac56-1632216a7d74","Type":"ContainerStarted","Data":"aec8bb94d4b544f82ee3a4a0060fc347c14a62f96689b149c472e9ff30837c16"}
Oct 30 19:21:13.779456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:13.779050    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" event={"ID":"527bd7b1-a500-4ea2-ac56-1632216a7d74","Type":"ContainerStarted","Data":"4857a4c76ac083a6522bafcee17322ac69cd84e6fbfd381f1333a75013bdd0fa"}
Oct 30 19:21:13.780649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:13.780626    4925 generic.go:356] "Generic (PLEG): container finished" podID="c5487022-9741-44d0-961b-3b60fe18fd78" containerID="d9ef8f481dc431a41791ebe2a08951f105265656b8046f33571d9d28cbd5312b" exitCode=0
Oct 30 19:21:13.780756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:13.780678    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb" event={"ID":"c5487022-9741-44d0-961b-3b60fe18fd78","Type":"ContainerDied","Data":"d9ef8f481dc431a41791ebe2a08951f105265656b8046f33571d9d28cbd5312b"}
Oct 30 19:21:13.801457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:13.801412    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podStartSLOduration=1.80139983 podStartE2EDuration="1.80139983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:13.799798629 +0000 UTC m=+908.702811663" watchObservedRunningTime="2026-10-30 19:21:13.80139983 +0000 UTC m=+908.704412867"
Oct 30 19:21:14.146779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.146744    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:14.645971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.645928    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:14.836500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.836464    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:14.958066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.958042    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:14.993763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.993738    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-xks-ca\") pod \"c5487022-9741-44d0-961b-3b60fe18fd78\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") "
Oct 30 19:21:14.993946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.993801    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5487022-9741-44d0-961b-3b60fe18fd78-kube-api-access-pkbdw\" (UniqueName: \"kubernetes.io/projected/c5487022-9741-44d0-961b-3b60fe18fd78-kube-api-access-pkbdw\") pod \"c5487022-9741-44d0-961b-3b60fe18fd78\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") "
Oct 30 19:21:14.993946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.993889    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-openshift-ca\") pod \"c5487022-9741-44d0-961b-3b60fe18fd78\" (UID: \"c5487022-9741-44d0-961b-3b60fe18fd78\") "
Oct 30 19:21:14.994323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.994297    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-openshift-ca" pod "c5487022-9741-44d0-961b-3b60fe18fd78" (UID: "c5487022-9741-44d0-961b-3b60fe18fd78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:14.996671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.996639    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-xks-ca" pod "c5487022-9741-44d0-961b-3b60fe18fd78" (UID: "c5487022-9741-44d0-961b-3b60fe18fd78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:14.997263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:14.997242    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5487022-9741-44d0-961b-3b60fe18fd78-kube-api-access-pkbdw" pod "c5487022-9741-44d0-961b-3b60fe18fd78" (UID: "c5487022-9741-44d0-961b-3b60fe18fd78"). InnerVolumeSpecName "kube-api-access-pkbdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:15.095018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.094987    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:15.095018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.095017    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkbdw\" (UniqueName: \"kubernetes.io/projected/c5487022-9741-44d0-961b-3b60fe18fd78-kube-api-access-pkbdw\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:15.095209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.095035    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5487022-9741-44d0-961b-3b60fe18fd78-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:15.563237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.563142    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:15.793370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.793342    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"
Oct 30 19:21:15.793370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.793356    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb" event={"ID":"c5487022-9741-44d0-961b-3b60fe18fd78","Type":"ContainerDied","Data":"6ff147e5df37f0d2355d1557ff67b12a416e05269a6d8a72c4adc34d67541517"}
Oct 30 19:21:15.793569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.793428    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ff147e5df37f0d2355d1557ff67b12a416e05269a6d8a72c4adc34d67541517"
Oct 30 19:21:15.891030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.890535    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"]
Oct 30 19:21:15.892480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.892456    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c5487022-9741-44d0-961b-3b60fe18fd78" containerName="revoke-keys"
Oct 30 19:21:15.892635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.892621    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5487022-9741-44d0-961b-3b60fe18fd78" containerName="revoke-keys"
Oct 30 19:21:15.892921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.892905    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5487022-9741-44d0-961b-3b60fe18fd78" containerName="revoke-keys"
Oct 30 19:21:15.897891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.897842    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:15.906286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.906247    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e73920-396b-4a7d-9573-a9009d5f5888-tls-certs\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:15.906493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.906475    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-tmp-dir\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:15.906648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.906635    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-model-cache\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:15.906773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.906756    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpkhq\" (UniqueName: \"kubernetes.io/projected/d4e73920-396b-4a7d-9573-a9009d5f5888-kube-api-access-lpkhq\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:15.906896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.906882    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-dshm\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:15.907003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.906990    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-home\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:15.912341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.912321    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"]
Oct 30 19:21:15.912634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.912619    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad23c7\"/\"distinct-w0-ad23c7-kserve-self-signed-certs\""
Oct 30 19:21:15.996833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:15.996599    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:16.008030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.007996    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-home\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.008347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.008324    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e73920-396b-4a7d-9573-a9009d5f5888-tls-certs\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.008497 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.008481    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-tmp-dir\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.008624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.008347    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-home\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.008811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.008791    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-model-cache\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.008964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.008945    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkhq\" (UniqueName: \"kubernetes.io/projected/d4e73920-396b-4a7d-9573-a9009d5f5888-kube-api-access-lpkhq\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.011229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.011204    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-dshm\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.011336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.009348    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-model-cache\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.011336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.009031    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-tmp-dir\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.012430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.012407    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e73920-396b-4a7d-9573-a9009d5f5888-tls-certs\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.013988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.013965    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-dshm\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.023200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.023149    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpkhq\" (UniqueName: \"kubernetes.io/projected/d4e73920-396b-4a7d-9573-a9009d5f5888-kube-api-access-lpkhq\") pod \"distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") " pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.248766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.244897    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:16.304672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:16.304635    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:17.094962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:17.091215    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"]
Oct 30 19:21:17.480265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:17.480158    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:17.963621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:17.963579    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" event={"ID":"d4e73920-396b-4a7d-9573-a9009d5f5888","Type":"ContainerStarted","Data":"bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d"}
Oct 30 19:21:17.963837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:17.963637    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" event={"ID":"d4e73920-396b-4a7d-9573-a9009d5f5888","Type":"ContainerStarted","Data":"73bd250cebf7cb9c9d7ee917a11f1b3682e8de9f8a7ad1a7b22de4057f1421e5"}
Oct 30 19:21:17.984334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:17.984282    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podStartSLOduration=2.984263481 podStartE2EDuration="2.984263481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:17.980816296 +0000 UTC m=+912.883829336" watchObservedRunningTime="2026-10-30 19:21:17.984263481 +0000 UTC m=+912.887276520"
Oct 30 19:21:18.246315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:18.246194    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:18.495069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:18.495031    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:19.478402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:19.478366    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:20.351790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:20.351756    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:21.056606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:21.056573    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:21.784366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:21.784334    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:22.125600 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:21:22.125523    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:21:22.646182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:22.646150    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:22.658882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:22.658857    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:22.865188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:22.865156    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:23.003999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:23.003929    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:21:23.821156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:23.821124    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:24.468622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:24.468586    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:25.358228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:25.358180    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:26.037223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:26.037188    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:26.245417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:26.245389    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:26.258054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:26.258032    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:26.856893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:26.856848    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:27.029098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:27.029072    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:21:27.502364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:27.502333    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:27.725324 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:27.725278    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:28.556747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:28.556716    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:29.626713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:29.623841    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:30.493713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:30.493431    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:31.656105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:31.656061    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:32.570035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:32.569991    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:33.178251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:33.178220    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:33.958900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:33.958865    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:34.565141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:34.565108    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:35.481309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:35.481278    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:36.186894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.186847    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:36.732462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.732429    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"]
Oct 30 19:21:36.738219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.738192    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.741528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.741505    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cfa4bb\"/\"distinct-2-w1-cfa4bb-kserve-self-signed-certs\""
Oct 30 19:21:36.746149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.746075    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"]
Oct 30 19:21:36.829975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.829935    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7cl\" (UniqueName: \"kubernetes.io/projected/d434c826-6f85-416c-8270-ccd49d8e38e9-kube-api-access-4r7cl\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.830195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.830102    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-home\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.830195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.830132    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-dshm\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.830315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.830227    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-model-cache\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.830378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.830310    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d434c826-6f85-416c-8270-ccd49d8e38e9-tls-certs\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.830428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.830376    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-tmp-dir\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.850353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.850317    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:36.931858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.931819    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-model-cache\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.931858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.931856    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d434c826-6f85-416c-8270-ccd49d8e38e9-tls-certs\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.932123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.931896    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-tmp-dir\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.932123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.932020    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7cl\" (UniqueName: \"kubernetes.io/projected/d434c826-6f85-416c-8270-ccd49d8e38e9-kube-api-access-4r7cl\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.932228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.932122    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-home\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.932228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.932151    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-dshm\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.932339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.932315    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-model-cache\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.932671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.932640    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-home\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.932846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.932818    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-tmp-dir\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.934988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.934961    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-dshm\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.935107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.935078    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d434c826-6f85-416c-8270-ccd49d8e38e9-tls-certs\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:36.945164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:36.945142    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7cl\" (UniqueName: \"kubernetes.io/projected/d434c826-6f85-416c-8270-ccd49d8e38e9-kube-api-access-4r7cl\") pod \"distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") " pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:37.052748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.052663    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:37.341147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.341008    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"]
Oct 30 19:21:37.344012 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:21:37.343980    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd434c826_6f85_416c_8270_ccd49d8e38e9.slice/crio-b68dbadb146cf592d0d98f6048bc2c3cef3c9e98055a52a340cedebc45af780a WatchSource:0}: Error finding container b68dbadb146cf592d0d98f6048bc2c3cef3c9e98055a52a340cedebc45af780a: Status 404 returned error can't find the container with id b68dbadb146cf592d0d98f6048bc2c3cef3c9e98055a52a340cedebc45af780a
Oct 30 19:21:37.387387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.387361    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:37.823615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.823582    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"]
Oct 30 19:21:37.827491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.827473    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:37.846344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.846321    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"]
Oct 30 19:21:37.944912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.944872    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:37.945036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.944994    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6h2\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-kube-api-access-hd6h2\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:37.945036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.945019    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:37.945109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.945053    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51928b0b-54d8-4959-b115-7f091a8c8700-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:37.945199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.945185    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:37.945254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.945239    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:37.945312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.945301    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:37.945350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.945320    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51928b0b-54d8-4959-b115-7f091a8c8700-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:37.945398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:37.945355    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.046507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046478    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6h2\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-kube-api-access-hd6h2\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.046507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046511    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.046795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046535    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51928b0b-54d8-4959-b115-7f091a8c8700-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.046795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046602    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.046795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046640    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.046795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046658    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.047061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046815    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51928b0b-54d8-4959-b115-7f091a8c8700-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.047061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046883    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.047061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046912    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.047061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.046957    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.047061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.047025    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.047315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.047271    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.047315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.047292    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.047524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.047504    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51928b0b-54d8-4959-b115-7f091a8c8700-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.049126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.049106    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.049237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.049220    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51928b0b-54d8-4959-b115-7f091a8c8700-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.059786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.059762    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6h2\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-kube-api-access-hd6h2\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.061546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.061527    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-v9tpn\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.111804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.111719    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" event={"ID":"d434c826-6f85-416c-8270-ccd49d8e38e9","Type":"ContainerStarted","Data":"2c9cd76ed50278c205261994beae1e72b6a582bd7554c8e66b7c7b5703ba9f31"}
Oct 30 19:21:38.111804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.111774    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" event={"ID":"d434c826-6f85-416c-8270-ccd49d8e38e9","Type":"ContainerStarted","Data":"b68dbadb146cf592d0d98f6048bc2c3cef3c9e98055a52a340cedebc45af780a"}
Oct 30 19:21:38.130491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.130451    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podStartSLOduration=2.130438145 podStartE2EDuration="2.130438145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:38.129118988 +0000 UTC m=+933.032132024" watchObservedRunningTime="2026-10-30 19:21:38.130438145 +0000 UTC m=+933.033451184"
Oct 30 19:21:38.141073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.141047    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:38.453610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.453584    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"]
Oct 30 19:21:38.456212 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:21:38.456173    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51928b0b_54d8_4959_b115_7f091a8c8700.slice/crio-50fcf05bed931432b2b18ffd8715d63e55255f2c7b854c2501738533f4d12676 WatchSource:0}: Error finding container 50fcf05bed931432b2b18ffd8715d63e55255f2c7b854c2501738533f4d12676: Status 404 returned error can't find the container with id 50fcf05bed931432b2b18ffd8715d63e55255f2c7b854c2501738533f4d12676
Oct 30 19:21:38.459226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.459192    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:38.459316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.459260    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:38.459316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.459291    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:38.621177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:38.621125    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:39.052969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:39.052942    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:39.127234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:39.127196    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" event={"ID":"51928b0b-54d8-4959-b115-7f091a8c8700","Type":"ContainerStarted","Data":"49da8472459ae0646c130ad78f38609571c853c61bfb62c60a34b1242977fe47"}
Oct 30 19:21:39.127433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:39.127258    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" event={"ID":"51928b0b-54d8-4959-b115-7f091a8c8700","Type":"ContainerStarted","Data":"50fcf05bed931432b2b18ffd8715d63e55255f2c7b854c2501738533f4d12676"}
Oct 30 19:21:39.141501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:39.141475    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:39.143247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:39.143225    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v9tpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:21:39.143355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:39.143265    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:21:39.151014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:39.150975    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podStartSLOduration=2.150962238 podStartE2EDuration="2.150962238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:39.148284717 +0000 UTC m=+934.051297732" watchObservedRunningTime="2026-10-30 19:21:39.150962238 +0000 UTC m=+934.053975274"
Oct 30 19:21:39.572460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:39.572420    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:40.145713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:40.144033    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v9tpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:21:40.145713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:40.144094    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:21:40.979501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:40.979465    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:41.142659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:41.142614    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v9tpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:21:41.142902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:41.142674    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:21:41.839807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:41.837246    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:42.142354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:42.142317    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v9tpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:21:42.142936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:42.142897    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:21:42.532329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:42.531730    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:43.141982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:43.141944    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v9tpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:21:43.142155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:43.142006    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:21:43.487965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:43.487885    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:44.064736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:44.064679    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:44.142062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:44.142032    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v9tpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:21:44.142243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:44.142077    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:21:44.668806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:44.668765    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:45.142645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:45.142617    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v9tpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:21:45.142834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:45.142666    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:21:45.572005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:45.571916    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:46.142824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.142738    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-v9tpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:21:46.143243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.142821    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:21:46.241678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.241646    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:46.634724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.634635    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"]
Oct 30 19:21:46.639600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.639578    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.651799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.651775    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"]
Oct 30 19:21:46.681541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.681511    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10779f52-qvc2r\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.681808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.681784    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10779f52-qvc2r\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.681902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.681844    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxmn\" (UniqueName: \"kubernetes.io/projected/2303cb33-dadf-4461-afbe-c3d1ede0bde6-kube-api-access-qbxmn\") pod \"maas-api-revoke-sub-10779f52-qvc2r\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.686353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.686328    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:46.714712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.711353    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:46.714712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.711434    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:21:46.791604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.790674    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10779f52-qvc2r\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.791604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.790771    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxmn\" (UniqueName: \"kubernetes.io/projected/2303cb33-dadf-4461-afbe-c3d1ede0bde6-kube-api-access-qbxmn\") pod \"maas-api-revoke-sub-10779f52-qvc2r\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.791604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.790849    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10779f52-qvc2r\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.797745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.793454    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10779f52-qvc2r\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.799108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.799080    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10779f52-qvc2r\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.808485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.808458    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxmn\" (UniqueName: \"kubernetes.io/projected/2303cb33-dadf-4461-afbe-c3d1ede0bde6-kube-api-access-qbxmn\") pod \"maas-api-revoke-sub-10779f52-qvc2r\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:46.976680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:46.976639    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:47.058935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:47.058908    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:47.077408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:47.077384    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:47.188353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:47.188307    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:47.188925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:47.188894    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:47.189881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:47.189860    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:21:47.194411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:47.194392    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:47.223484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:47.223364    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:21:47.277791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:47.277762    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"]
Oct 30 19:21:47.286231 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:21:47.286199    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2303cb33_dadf_4461_afbe_c3d1ede0bde6.slice/crio-51413050295ac9bdc22ff47ca53387888a229c0392f4c1d0f5644bece5de8245 WatchSource:0}: Error finding container 51413050295ac9bdc22ff47ca53387888a229c0392f4c1d0f5644bece5de8245: Status 404 returned error can't find the container with id 51413050295ac9bdc22ff47ca53387888a229c0392f4c1d0f5644bece5de8245
Oct 30 19:21:47.955639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:47.955605    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:48.219165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:48.219142    4925 generic.go:356] "Generic (PLEG): container finished" podID="2303cb33-dadf-4461-afbe-c3d1ede0bde6" containerID="1d4eb6c8d0658cd22dad3c90f721669ab1a4fed3620086c207ef8e4589d51c37" exitCode=0
Oct 30 19:21:48.219467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:48.219227    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r" event={"ID":"2303cb33-dadf-4461-afbe-c3d1ede0bde6","Type":"ContainerDied","Data":"1d4eb6c8d0658cd22dad3c90f721669ab1a4fed3620086c207ef8e4589d51c37"}
Oct 30 19:21:48.219467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:48.219280    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r" event={"ID":"2303cb33-dadf-4461-afbe-c3d1ede0bde6","Type":"ContainerStarted","Data":"51413050295ac9bdc22ff47ca53387888a229c0392f4c1d0f5644bece5de8245"}
Oct 30 19:21:48.656287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:48.656255    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:49.272354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.272326    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:49.393852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.393828    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:49.423101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.423076    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2303cb33-dadf-4461-afbe-c3d1ede0bde6-kube-api-access-qbxmn\" (UniqueName: \"kubernetes.io/projected/2303cb33-dadf-4461-afbe-c3d1ede0bde6-kube-api-access-qbxmn\") pod \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") "
Oct 30 19:21:49.423262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.423123    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-openshift-ca\") pod \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") "
Oct 30 19:21:49.423262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.423194    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-xks-ca\") pod \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\" (UID: \"2303cb33-dadf-4461-afbe-c3d1ede0bde6\") "
Oct 30 19:21:49.423523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.423501    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-openshift-ca" pod "2303cb33-dadf-4461-afbe-c3d1ede0bde6" (UID: "2303cb33-dadf-4461-afbe-c3d1ede0bde6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:49.425531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.425503    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-xks-ca" pod "2303cb33-dadf-4461-afbe-c3d1ede0bde6" (UID: "2303cb33-dadf-4461-afbe-c3d1ede0bde6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:49.425531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.425504    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2303cb33-dadf-4461-afbe-c3d1ede0bde6-kube-api-access-qbxmn" pod "2303cb33-dadf-4461-afbe-c3d1ede0bde6" (UID: "2303cb33-dadf-4461-afbe-c3d1ede0bde6"). InnerVolumeSpecName "kube-api-access-qbxmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:49.525067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.524993    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:49.525067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.525020    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbxmn\" (UniqueName: \"kubernetes.io/projected/2303cb33-dadf-4461-afbe-c3d1ede0bde6-kube-api-access-qbxmn\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:49.525067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.525031    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2303cb33-dadf-4461-afbe-c3d1ede0bde6-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:49.629096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.629066    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"]
Oct 30 19:21:49.629636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.629617    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2303cb33-dadf-4461-afbe-c3d1ede0bde6" containerName="revoke-keys"
Oct 30 19:21:49.629636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.629637    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2303cb33-dadf-4461-afbe-c3d1ede0bde6" containerName="revoke-keys"
Oct 30 19:21:49.629786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.629767    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2303cb33-dadf-4461-afbe-c3d1ede0bde6" containerName="revoke-keys"
Oct 30 19:21:49.633354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.633329    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.646874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.646849    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"]
Oct 30 19:21:49.727406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.727298    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.727406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.727340    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk88n\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-kube-api-access-qk88n\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.727624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.727469    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.727624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.727531    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.727624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.727559    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.727624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.727578    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.727812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.727720    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.727812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.727749    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.727812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.727766    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.826665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.826593    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"]
Oct 30 19:21:49.829160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829138    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829171    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk88n\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-kube-api-access-qk88n\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829216    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829486    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829522    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829545    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829524    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829597    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829726    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829767    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829792    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829841    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.829914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.829848    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.830452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.830435    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.830509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.830452    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.832378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.832359    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.832669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.832649    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.844224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.844204    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk88n\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-kube-api-access-qk88n\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.845354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.845227    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.849850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.849827    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"]
Oct 30 19:21:49.870428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.870400    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"]
Oct 30 19:21:49.875167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.875143    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.895280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.886949    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"]
Oct 30 19:21:49.930911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.930879    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.930911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.930912    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.931126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.930938    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfe234f-50c6-4823-abb3-b07e22e285dd-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.931126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931011    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.931126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931113    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.931249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931140    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5dc9\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-kube-api-access-v5dc9\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.931249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931166    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.931325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931251    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.931325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931280    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.931325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931314    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.931428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931392    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.931428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931412    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.931524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931443    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.931524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931479    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.931632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931548    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ee6e4e-8653-4307-a279-f0528a548aef-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.931632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931590    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:49.931632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931616    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.931788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.931640    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29qd\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-kube-api-access-x29qd\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:49.948769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.948736    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:49.994127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:49.994025    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:50.033012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.032983    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.033012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033016    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.033219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033038    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29qd\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-kube-api-access-x29qd\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.033219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033085    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.033219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033114    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.033219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033150    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfe234f-50c6-4823-abb3-b07e22e285dd-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.033219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033185    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.033219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033213    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.033527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033239    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5dc9\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-kube-api-access-v5dc9\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.033527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033276    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.033527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033328    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.033527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033397    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.033527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033447    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.033818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033543    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.033818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033572    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.033818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033631    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.033818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033664    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.033818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.033713    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ee6e4e-8653-4307-a279-f0528a548aef-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.034523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.034501    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.034958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.034930    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.034958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.034579    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.035176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.034983    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.035176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.034508    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ee6e4e-8653-4307-a279-f0528a548aef-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.035176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.035024    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-credential-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.035176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.035051    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-certs\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.035393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.035223    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-socket\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.035393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.035246    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-data\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.035643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.035622    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfe234f-50c6-4823-abb3-b07e22e285dd-istiod-ca-cert\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.037167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.037147    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.038102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.038074    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-podinfo\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.038468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.038441    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.039712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.039678    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-envoy\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.045609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.045322    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5dc9\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-kube-api-access-v5dc9\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.045609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.045582    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.046157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.046136    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-token\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.046253 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.046239    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29qd\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-kube-api-access-x29qd\") pod \"e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") " pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.160744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.160714    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:50.191794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.191762    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:50.238351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.236505    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"
Oct 30 19:21:50.238351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.236535    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r" event={"ID":"2303cb33-dadf-4461-afbe-c3d1ede0bde6","Type":"ContainerDied","Data":"51413050295ac9bdc22ff47ca53387888a229c0392f4c1d0f5644bece5de8245"}
Oct 30 19:21:50.238351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.236607    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51413050295ac9bdc22ff47ca53387888a229c0392f4c1d0f5644bece5de8245"
Oct 30 19:21:50.247042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.246761    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"]
Oct 30 19:21:50.266706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.265886    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:50.266706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.265980    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:50.266706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.266030    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:50.465202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.465172    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"]
Oct 30 19:21:50.466664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:21:50.466636    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfe234f_50c6_4823_abb3_b07e22e285dd.slice/crio-844946b0c7b9a74b83c71909aa8f2efbdb5bdfedd94b4f2d96865eaca9b5225f WatchSource:0}: Error finding container 844946b0c7b9a74b83c71909aa8f2efbdb5bdfedd94b4f2d96865eaca9b5225f: Status 404 returned error can't find the container with id 844946b0c7b9a74b83c71909aa8f2efbdb5bdfedd94b4f2d96865eaca9b5225f
Oct 30 19:21:50.469410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.469367    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:50.469516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.469439    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:50.469516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.469474    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:50.511369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.511346    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"]
Oct 30 19:21:50.518925 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:21:50.518891    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ee6e4e_8653_4307_a279_f0528a548aef.slice/crio-2b5e60ded537536a45c9169017d81f6f692fdeb9c98086ae4bfa61268bc26805 WatchSource:0}: Error finding container 2b5e60ded537536a45c9169017d81f6f692fdeb9c98086ae4bfa61268bc26805: Status 404 returned error can't find the container with id 2b5e60ded537536a45c9169017d81f6f692fdeb9c98086ae4bfa61268bc26805
Oct 30 19:21:50.521953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.521914    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:50.522048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.522003    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:50.522089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:50.522048    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:21:51.211329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.211298    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:51.245187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.245149    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" event={"ID":"e3ee6e4e-8653-4307-a279-f0528a548aef","Type":"ContainerStarted","Data":"e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8"}
Oct 30 19:21:51.245366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.245207    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" event={"ID":"e3ee6e4e-8653-4307-a279-f0528a548aef","Type":"ContainerStarted","Data":"2b5e60ded537536a45c9169017d81f6f692fdeb9c98086ae4bfa61268bc26805"}
Oct 30 19:21:51.246958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.246931    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" event={"ID":"03d906b5-98ed-42f0-8ea7-3ca948f57e7b","Type":"ContainerStarted","Data":"ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d"}
Oct 30 19:21:51.247100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.246978    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" event={"ID":"03d906b5-98ed-42f0-8ea7-3ca948f57e7b","Type":"ContainerStarted","Data":"2d1ff58f12e747587f004d5289014f93e16acca81ac69556df36ad4652bc82f7"}
Oct 30 19:21:51.248547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.248524    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" event={"ID":"6cfe234f-50c6-4823-abb3-b07e22e285dd","Type":"ContainerStarted","Data":"645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b"}
Oct 30 19:21:51.248655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.248563    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" event={"ID":"6cfe234f-50c6-4823-abb3-b07e22e285dd","Type":"ContainerStarted","Data":"844946b0c7b9a74b83c71909aa8f2efbdb5bdfedd94b4f2d96865eaca9b5225f"}
Oct 30 19:21:51.275680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.275637    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" podStartSLOduration=2.2756231639999998 podStartE2EDuration="2.275623164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:51.272248911 +0000 UTC m=+946.175261948" watchObservedRunningTime="2026-10-30 19:21:51.275623164 +0000 UTC m=+946.178636195"
Oct 30 19:21:51.348718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.348648    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podStartSLOduration=2.348633149 podStartE2EDuration="2.348633149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:51.347452582 +0000 UTC m=+946.250465619" watchObservedRunningTime="2026-10-30 19:21:51.348633149 +0000 UTC m=+946.251646185"
Oct 30 19:21:51.348880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.348792    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" podStartSLOduration=2.348785005 podStartE2EDuration="2.348785005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:51.319737578 +0000 UTC m=+946.222750618" watchObservedRunningTime="2026-10-30 19:21:51.348785005 +0000 UTC m=+946.251798044"
Oct 30 19:21:51.953468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.951796    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:21:51.953468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.952006    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:51.953468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:51.952057    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:21:52.164718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:52.163554    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:52.186743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:52.177188    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:52.186743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:52.177272    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 19:21:52.197212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:52.195858    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:52.197540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:52.197518    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:21:52.197747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:52.197721    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:21:52.198800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:52.198778    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:52.267771 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:21:52.264518    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:21:52.953713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:52.953255    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:52.953713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:52.953317    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:21:53.150637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:53.148628    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:53.162075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:53.162044    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:53.162340 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:53.162310    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 19:21:53.209773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:53.206473    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:21:53.209773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:53.206545    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:21:53.952713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:53.952161    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:53.952713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:53.952224    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:21:54.172755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:54.170062    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:54.172755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:54.170123    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 19:21:54.192309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:54.192276    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:21:54.192473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:54.192330    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:21:54.530118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:54.530084    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:54.953592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:54.953113    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:54.953592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:54.953176    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:21:55.170770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:55.169933    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:55.170770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:55.169990    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 19:21:55.192784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:55.192748    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:21:55.194483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:55.194428    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:21:55.853717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:55.853069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:55.952122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:55.952084    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:55.952419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:55.952384    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:21:56.163225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:56.163194    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:56.163525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:56.163484    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 19:21:56.193137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:56.193102    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:21:56.193724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:56.193666    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:21:56.521774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:56.520671    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:56.950898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:56.950862    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:56.951143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:56.951118    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:21:57.165717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:57.162001    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:57.165717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:57.162059    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 19:21:57.195036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:57.195003    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:21:57.195593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:57.195568    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:21:57.950715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:57.950124    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:21:57.950715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:57.950191    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:21:58.162394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:58.161953    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:21:58.162394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:58.162010    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 19:21:58.195485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:58.195039    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:21:58.195485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:58.195096    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 19:21:58.984779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:58.984742    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:59.276808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:59.274929    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:59.276808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:59.275993    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:59.282964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:59.282900    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:21:59.584573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:59.584488    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:59.586232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:59.586212    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:59.594978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:59.594952    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:21:59.990836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:59.990797    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:21:59.991025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:21:59.990862    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:00.530019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:00.529984    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:22:00.530506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:00.530389    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:22:00.531284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:00.531264    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:22:01.203116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.203078    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"]
Oct 30 19:22:01.207269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.207251    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.215707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.214747    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:01.215707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.215129    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:22:01.221105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.221079    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"]
Oct 30 19:22:01.311675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.311640    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa3296ac-xq4nd\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.311884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.311699    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhsm\" (UniqueName: \"kubernetes.io/projected/26a4d7df-1431-44a1-8718-98a44bd9561c-kube-api-access-vwhsm\") pod \"maas-api-revoke-sub-aa3296ac-xq4nd\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.312006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.311986    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa3296ac-xq4nd\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.413287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.413256    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa3296ac-xq4nd\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.413488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.413316    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa3296ac-xq4nd\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.413488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.413339    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhsm\" (UniqueName: \"kubernetes.io/projected/26a4d7df-1431-44a1-8718-98a44bd9561c-kube-api-access-vwhsm\") pod \"maas-api-revoke-sub-aa3296ac-xq4nd\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.413987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.413963    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa3296ac-xq4nd\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.415956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.415933    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa3296ac-xq4nd\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.425404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.425381    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhsm\" (UniqueName: \"kubernetes.io/projected/26a4d7df-1431-44a1-8718-98a44bd9561c-kube-api-access-vwhsm\") pod \"maas-api-revoke-sub-aa3296ac-xq4nd\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.525194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.525121    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:01.808132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:01.808107    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"]
Oct 30 19:22:01.809928 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:22:01.809902    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a4d7df_1431_44a1_8718_98a44bd9561c.slice/crio-b12cd3b3e723dd0205994534e8a4bdd12af69809d79ee383519cd20fedecbba0 WatchSource:0}: Error finding container b12cd3b3e723dd0205994534e8a4bdd12af69809d79ee383519cd20fedecbba0: Status 404 returned error can't find the container with id b12cd3b3e723dd0205994534e8a4bdd12af69809d79ee383519cd20fedecbba0
Oct 30 19:22:02.052663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:02.052626    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:02.580587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:02.580552    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd" event={"ID":"26a4d7df-1431-44a1-8718-98a44bd9561c","Type":"ContainerStarted","Data":"31e45d6822162f03eebdcf060a593dbb7c20b813312f9afae62c92884962daa7"}
Oct 30 19:22:02.580769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:02.580608    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd" event={"ID":"26a4d7df-1431-44a1-8718-98a44bd9561c","Type":"ContainerStarted","Data":"b12cd3b3e723dd0205994534e8a4bdd12af69809d79ee383519cd20fedecbba0"}
Oct 30 19:22:02.598792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:02.598747    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd" podStartSLOduration=1.5987327169999999 podStartE2EDuration="1.598732717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:02.59624118 +0000 UTC m=+957.499254208" watchObservedRunningTime="2026-10-30 19:22:02.598732717 +0000 UTC m=+957.501745754"
Oct 30 19:22:03.475396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:03.475354    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:03.589569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:03.589533    4925 generic.go:356] "Generic (PLEG): container finished" podID="26a4d7df-1431-44a1-8718-98a44bd9561c" containerID="31e45d6822162f03eebdcf060a593dbb7c20b813312f9afae62c92884962daa7" exitCode=0
Oct 30 19:22:03.589750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:03.589590    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd" event={"ID":"26a4d7df-1431-44a1-8718-98a44bd9561c","Type":"ContainerDied","Data":"31e45d6822162f03eebdcf060a593dbb7c20b813312f9afae62c92884962daa7"}
Oct 30 19:22:04.506608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.506570    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:04.740350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.740327    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:04.855540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.855463    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a4d7df-1431-44a1-8718-98a44bd9561c-kube-api-access-vwhsm\" (UniqueName: \"kubernetes.io/projected/26a4d7df-1431-44a1-8718-98a44bd9561c-kube-api-access-vwhsm\") pod \"26a4d7df-1431-44a1-8718-98a44bd9561c\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") "
Oct 30 19:22:04.855540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.855531    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-openshift-ca\") pod \"26a4d7df-1431-44a1-8718-98a44bd9561c\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") "
Oct 30 19:22:04.855767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.855563    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-xks-ca\") pod \"26a4d7df-1431-44a1-8718-98a44bd9561c\" (UID: \"26a4d7df-1431-44a1-8718-98a44bd9561c\") "
Oct 30 19:22:04.855982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.855958    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-openshift-ca" pod "26a4d7df-1431-44a1-8718-98a44bd9561c" (UID: "26a4d7df-1431-44a1-8718-98a44bd9561c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:04.857844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.857816    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-xks-ca" pod "26a4d7df-1431-44a1-8718-98a44bd9561c" (UID: "26a4d7df-1431-44a1-8718-98a44bd9561c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:04.857844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.857837    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a4d7df-1431-44a1-8718-98a44bd9561c-kube-api-access-vwhsm" pod "26a4d7df-1431-44a1-8718-98a44bd9561c" (UID: "26a4d7df-1431-44a1-8718-98a44bd9561c"). InnerVolumeSpecName "kube-api-access-vwhsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:04.957087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.957056    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhsm\" (UniqueName: \"kubernetes.io/projected/26a4d7df-1431-44a1-8718-98a44bd9561c-kube-api-access-vwhsm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:04.957087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.957083    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:04.957087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:04.957095    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a4d7df-1431-44a1-8718-98a44bd9561c-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:05.183180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.183149    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"]
Oct 30 19:22:05.183634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.183622    4925 cpu_manager.go:414] "containerMap: removing container" podUID="26a4d7df-1431-44a1-8718-98a44bd9561c" containerName="revoke-keys"
Oct 30 19:22:05.183724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.183636    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a4d7df-1431-44a1-8718-98a44bd9561c" containerName="revoke-keys"
Oct 30 19:22:05.183768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.183744    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a4d7df-1431-44a1-8718-98a44bd9561c" containerName="revoke-keys"
Oct 30 19:22:05.187039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.187017    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.200927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.200902    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"]
Oct 30 19:22:05.259448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.259418    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.259604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.259493    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.259604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.259526    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.259714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.259653    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.259750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.259731    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.259798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.259756    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.259840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.259827    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.259875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.259853    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4xnv\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-kube-api-access-t4xnv\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.260010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.259990    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361038    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361102    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361165    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361194    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361282    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361312    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361342    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361381    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361409    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361450    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4xnv\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-kube-api-access-t4xnv\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361520    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.361853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.361826    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.362195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.362165    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.362365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.362337    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.364117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.364094    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.364117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.364106    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.374593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.374569    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.374808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.374784    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4xnv\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-kube-api-access-t4xnv\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.428732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.428702    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:05.501157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.501067    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:05.606241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.606214    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"
Oct 30 19:22:05.606734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.606214    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd" event={"ID":"26a4d7df-1431-44a1-8718-98a44bd9561c","Type":"ContainerDied","Data":"b12cd3b3e723dd0205994534e8a4bdd12af69809d79ee383519cd20fedecbba0"}
Oct 30 19:22:05.606822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.606768    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b12cd3b3e723dd0205994534e8a4bdd12af69809d79ee383519cd20fedecbba0"
Oct 30 19:22:05.832308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.832280    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"]
Oct 30 19:22:05.833307 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:22:05.833282    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f8dcc4_3d54_4c3a_b104_00529b0dac80.slice/crio-3ea50a76c3d2348e366410c68d0fd97a78549608dd537ea9a96102cba105b7b5 WatchSource:0}: Error finding container 3ea50a76c3d2348e366410c68d0fd97a78549608dd537ea9a96102cba105b7b5: Status 404 returned error can't find the container with id 3ea50a76c3d2348e366410c68d0fd97a78549608dd537ea9a96102cba105b7b5
Oct 30 19:22:05.835827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.835797    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:05.835886 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.835859    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:05.835928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:05.835891    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:06.488423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:06.488391    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:06.631239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:06.631200    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" event={"ID":"e5f8dcc4-3d54-4c3a-b104-00529b0dac80","Type":"ContainerStarted","Data":"ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304"}
Oct 30 19:22:06.631725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:06.631296    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" event={"ID":"e5f8dcc4-3d54-4c3a-b104-00529b0dac80","Type":"ContainerStarted","Data":"3ea50a76c3d2348e366410c68d0fd97a78549608dd537ea9a96102cba105b7b5"}
Oct 30 19:22:06.656616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:06.656562    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podStartSLOduration=1.656546964 podStartE2EDuration="1.656546964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:06.654916713 +0000 UTC m=+961.557929750" watchObservedRunningTime="2026-10-30 19:22:06.656546964 +0000 UTC m=+961.559560001"
Oct 30 19:22:07.156669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:07.156637    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:07.516101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:07.516016    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:07.520917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:07.520872    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:22:07.521075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:07.520941    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:22:08.027426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:08.027394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:08.513070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:08.513033    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:22:08.513317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:08.513288    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:22:08.731710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:08.730461    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:09.384828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:09.384786    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:09.507715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:09.502016    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:22:09.507715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:09.502083    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:22:10.114713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.113590    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:10.334903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.334871    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"]
Oct 30 19:22:10.338916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.338900    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.342843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.342786    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:10.343396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.343372    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:22:10.346340 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.346323    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"]
Oct 30 19:22:10.451162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.451125    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb9f8cc-hj9fw\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.451162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.451161    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb9f8cc-hj9fw\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.451585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.451314    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ngv2\" (UniqueName: \"kubernetes.io/projected/40add373-e5a0-42f5-ba37-2b578dedba7f-kube-api-access-9ngv2\") pod \"maas-api-revoke-sub-1bb9f8cc-hj9fw\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.514711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.510074    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:22:10.514711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.510133    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:22:10.552912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.552875    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb9f8cc-hj9fw\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.552912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.552913    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb9f8cc-hj9fw\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.553138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.552958    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ngv2\" (UniqueName: \"kubernetes.io/projected/40add373-e5a0-42f5-ba37-2b578dedba7f-kube-api-access-9ngv2\") pod \"maas-api-revoke-sub-1bb9f8cc-hj9fw\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.553598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.553572    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb9f8cc-hj9fw\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.555467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.555447    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb9f8cc-hj9fw\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.569613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.569589    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ngv2\" (UniqueName: \"kubernetes.io/projected/40add373-e5a0-42f5-ba37-2b578dedba7f-kube-api-access-9ngv2\") pod \"maas-api-revoke-sub-1bb9f8cc-hj9fw\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.654501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.654468    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:10.952260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.952178    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:10.957870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:10.956518    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"]
Oct 30 19:22:11.512782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:11.512748    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:22:11.513353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:11.513318    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:22:11.663758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:11.663722    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw" event={"ID":"40add373-e5a0-42f5-ba37-2b578dedba7f","Type":"ContainerStarted","Data":"a1a6f8b2ef555e4571330eb950890ed2dffe8ba8f2c3429426290169d08a40a6"}
Oct 30 19:22:11.663991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:11.663778    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw" event={"ID":"40add373-e5a0-42f5-ba37-2b578dedba7f","Type":"ContainerStarted","Data":"2a2a66e4d249a7a15275d041a215a690c8260a4d9135e01cd07bda3bcfda0632"}
Oct 30 19:22:11.682321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:11.682266    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw" podStartSLOduration=1.682253419 podStartE2EDuration="1.682253419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:11.680497767 +0000 UTC m=+966.583510783" watchObservedRunningTime="2026-10-30 19:22:11.682253419 +0000 UTC m=+966.585266470"
Oct 30 19:22:11.776307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:11.776202    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:12.501760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:12.501730    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:22:12.501967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:12.501784    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 19:22:12.550073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:12.549969    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:12.672077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:12.672043    4925 generic.go:356] "Generic (PLEG): container finished" podID="40add373-e5a0-42f5-ba37-2b578dedba7f" containerID="a1a6f8b2ef555e4571330eb950890ed2dffe8ba8f2c3429426290169d08a40a6" exitCode=0
Oct 30 19:22:12.672257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:12.672093    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw" event={"ID":"40add373-e5a0-42f5-ba37-2b578dedba7f","Type":"ContainerDied","Data":"a1a6f8b2ef555e4571330eb950890ed2dffe8ba8f2c3429426290169d08a40a6"}
Oct 30 19:22:13.276814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.276779    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:13.505746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.505713    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:13.506123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.506101    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:13.506793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.506775    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:22:13.843811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.843785    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:13.892410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.892376    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:13.899000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.898971    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40add373-e5a0-42f5-ba37-2b578dedba7f-kube-api-access-9ngv2\" (UniqueName: \"kubernetes.io/projected/40add373-e5a0-42f5-ba37-2b578dedba7f-kube-api-access-9ngv2\") pod \"40add373-e5a0-42f5-ba37-2b578dedba7f\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") "
Oct 30 19:22:13.899175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.899048    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-xks-ca\") pod \"40add373-e5a0-42f5-ba37-2b578dedba7f\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") "
Oct 30 19:22:13.899175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.899121    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-openshift-ca\") pod \"40add373-e5a0-42f5-ba37-2b578dedba7f\" (UID: \"40add373-e5a0-42f5-ba37-2b578dedba7f\") "
Oct 30 19:22:13.900004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.899978    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-openshift-ca" pod "40add373-e5a0-42f5-ba37-2b578dedba7f" (UID: "40add373-e5a0-42f5-ba37-2b578dedba7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:13.919402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.919350    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40add373-e5a0-42f5-ba37-2b578dedba7f-kube-api-access-9ngv2" pod "40add373-e5a0-42f5-ba37-2b578dedba7f" (UID: "40add373-e5a0-42f5-ba37-2b578dedba7f"). InnerVolumeSpecName "kube-api-access-9ngv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:13.922184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:13.922153    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-xks-ca" pod "40add373-e5a0-42f5-ba37-2b578dedba7f" (UID: "40add373-e5a0-42f5-ba37-2b578dedba7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:14.000481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:14.000448    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ngv2\" (UniqueName: \"kubernetes.io/projected/40add373-e5a0-42f5-ba37-2b578dedba7f-kube-api-access-9ngv2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:14.000481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:14.000477    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:14.000481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:14.000489    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40add373-e5a0-42f5-ba37-2b578dedba7f-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:14.660862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:14.660828    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:14.692421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:14.692396    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"
Oct 30 19:22:14.692733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:14.692424    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw" event={"ID":"40add373-e5a0-42f5-ba37-2b578dedba7f","Type":"ContainerDied","Data":"2a2a66e4d249a7a15275d041a215a690c8260a4d9135e01cd07bda3bcfda0632"}
Oct 30 19:22:14.692879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:14.692754    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a2a66e4d249a7a15275d041a215a690c8260a4d9135e01cd07bda3bcfda0632"
Oct 30 19:22:15.467605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.467545    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:15.841771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.841657    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"]
Oct 30 19:22:15.842281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.842263    4925 cpu_manager.go:414] "containerMap: removing container" podUID="40add373-e5a0-42f5-ba37-2b578dedba7f" containerName="revoke-keys"
Oct 30 19:22:15.842382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.842282    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40add373-e5a0-42f5-ba37-2b578dedba7f" containerName="revoke-keys"
Oct 30 19:22:15.842488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.842475    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="40add373-e5a0-42f5-ba37-2b578dedba7f" containerName="revoke-keys"
Oct 30 19:22:15.846024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.846004    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:15.852366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.852342    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39b867\"/\"distinct-w2-39b867-kserve-self-signed-certs\""
Oct 30 19:22:15.856136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.856113    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"]
Oct 30 19:22:15.925305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.925266    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-model-cache\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:15.925483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.925313    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-dshm\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:15.925551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.925511    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzwl\" (UniqueName: \"kubernetes.io/projected/a3234780-a8fa-4d2c-8717-01d5d186506f-kube-api-access-tlzwl\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:15.925753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.925681    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-home\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:15.925853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.925751    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-tmp-dir\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:15.925853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.925786    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3234780-a8fa-4d2c-8717-01d5d186506f-tls-certs\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:15.945516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:15.945477    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:16.031665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.031628    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-home\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.031957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.031934    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-tmp-dir\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.032074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.032061    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3234780-a8fa-4d2c-8717-01d5d186506f-tls-certs\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.032207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.032192    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-model-cache\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.032321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.032309    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-dshm\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.032568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.032548    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzwl\" (UniqueName: \"kubernetes.io/projected/a3234780-a8fa-4d2c-8717-01d5d186506f-kube-api-access-tlzwl\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.033504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.033482    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-model-cache\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.033773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.033747    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-home\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.034307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.034004    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-tmp-dir\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.048514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.048204    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-dshm\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.049813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.049789    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3234780-a8fa-4d2c-8717-01d5d186506f-tls-certs\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.057558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.052672    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzwl\" (UniqueName: \"kubernetes.io/projected/a3234780-a8fa-4d2c-8717-01d5d186506f-kube-api-access-tlzwl\") pod \"distinct-w2-39b867-kserve-7fc56f95bb-xfb85\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.160533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.160506    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:16.465751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.456097    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:16.466633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.466605    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"]
Oct 30 19:22:16.474615 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:22:16.474587    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3234780_a8fa_4d2c_8717_01d5d186506f.slice/crio-59350e01da421c4697a4ccd147959b0f2cc492680c7fb54d2eb6b636d852c530 WatchSource:0}: Error finding container 59350e01da421c4697a4ccd147959b0f2cc492680c7fb54d2eb6b636d852c530: Status 404 returned error can't find the container with id 59350e01da421c4697a4ccd147959b0f2cc492680c7fb54d2eb6b636d852c530
Oct 30 19:22:16.705104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.705072    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" event={"ID":"a3234780-a8fa-4d2c-8717-01d5d186506f","Type":"ContainerStarted","Data":"01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af"}
Oct 30 19:22:16.705283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.705119    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" event={"ID":"a3234780-a8fa-4d2c-8717-01d5d186506f","Type":"ContainerStarted","Data":"59350e01da421c4697a4ccd147959b0f2cc492680c7fb54d2eb6b636d852c530"}
Oct 30 19:22:16.723056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:16.722967    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podStartSLOduration=1.722953841 podStartE2EDuration="1.722953841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:16.722641587 +0000 UTC m=+971.625654676" watchObservedRunningTime="2026-10-30 19:22:16.722953841 +0000 UTC m=+971.625966878"
Oct 30 19:22:17.161263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:17.161234    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:17.275215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:17.275152    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:18.106791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:18.106757    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:19.188009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.187981    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:19.739649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.739611    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"]
Oct 30 19:22:19.744274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.744242    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.764115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.764090    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"]
Oct 30 19:22:19.891940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.891910    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6792554-663c-4785-940c-da31095bd57f-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.892102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.891958    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-565tw\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-kube-api-access-565tw\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.892102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.892082    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.892177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.892119    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.892210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.892183    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.892244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.892216    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.892244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.892236    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6792554-663c-4785-940c-da31095bd57f-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.892304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.892278    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.892371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.892357    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.970247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.970204    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:19.993912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.993824    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6792554-663c-4785-940c-da31095bd57f-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.993912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.993903    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-565tw\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-kube-api-access-565tw\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.993972    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994009    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994051    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994090    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994121    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6792554-663c-4785-940c-da31095bd57f-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994188    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994255    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994502    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994578    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6792554-663c-4785-940c-da31095bd57f-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994625    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.994774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.994749    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.995039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.995015    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.996513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.996495    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:19.997055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:19.997031    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6792554-663c-4785-940c-da31095bd57f-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:20.005806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.005773    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:20.006568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.006544    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-565tw\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-kube-api-access-565tw\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:20.067226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.067195    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:20.358336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.358313    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"]
Oct 30 19:22:20.360184 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:22:20.360156    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6792554_663c_4785_940c_da31095bd57f.slice/crio-0ef509ac95fb476f0ccea07a7fe623428c7ace0c3325eebe1bc7a92ecfa15156 WatchSource:0}: Error finding container 0ef509ac95fb476f0ccea07a7fe623428c7ace0c3325eebe1bc7a92ecfa15156: Status 404 returned error can't find the container with id 0ef509ac95fb476f0ccea07a7fe623428c7ace0c3325eebe1bc7a92ecfa15156
Oct 30 19:22:20.362638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.362607    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:20.362733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.362709    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:20.362778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.362746    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:20.737617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.737575    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" event={"ID":"f6792554-663c-4785-940c-da31095bd57f","Type":"ContainerStarted","Data":"3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8"}
Oct 30 19:22:20.737832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.737635    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" event={"ID":"f6792554-663c-4785-940c-da31095bd57f","Type":"ContainerStarted","Data":"0ef509ac95fb476f0ccea07a7fe623428c7ace0c3325eebe1bc7a92ecfa15156"}
Oct 30 19:22:20.738279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.738237    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:20.759454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:20.759404    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podStartSLOduration=1.759386402 podStartE2EDuration="1.759386402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:20.758890664 +0000 UTC m=+975.661903702" watchObservedRunningTime="2026-10-30 19:22:20.759386402 +0000 UTC m=+975.662399439"
Oct 30 19:22:21.067631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:21.067545    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:21.069111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:21.069082    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:21.069228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:21.069136    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:22:21.444719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:21.444661    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:22.068296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:22.068260    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:22.068466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:22.068320    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:22:22.144729 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:22:22.144666    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:22:22.338812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:22.338714    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:23.067678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:23.067643    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:23.068177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:23.067707    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:22:23.248670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:23.248607    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:23.919225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:23.919185    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:24.067658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:24.067622    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:24.067845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:24.067718    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:22:24.669028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:24.668992    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:25.068477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:25.068382    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:25.068477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:25.068442    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:22:25.311026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:25.310996    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:25.929753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:25.926419    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:26.068160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:26.068125    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:26.068336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:26.068190    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:22:26.161195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:26.161162    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:26.175129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:26.175106    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:26.741281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:26.741251    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:26.789782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:26.789757    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:22:27.067470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:27.067391    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:22:27.067470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:27.067449    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 19:22:27.467333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:27.467290    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:28.133633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:28.133601    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:28.135019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:28.134972    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:28.136025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:28.136007    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:22:28.165055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:28.165029    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:28.966251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:28.966204    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:29.057436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.057409    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"]
Oct 30 19:22:29.062982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.062963    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.069523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.068933    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:29.070448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.070424    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:22:29.070633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.070593    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"]
Oct 30 19:22:29.225248 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.225163    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c1064da-zvcw6\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.225431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.225413    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7688\" (UniqueName: \"kubernetes.io/projected/4e96f89d-f177-43b9-847b-cc7caca2250d-kube-api-access-f7688\") pod \"maas-api-revoke-sub-2c1064da-zvcw6\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.225515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.225472    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c1064da-zvcw6\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.327202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.327173    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7688\" (UniqueName: \"kubernetes.io/projected/4e96f89d-f177-43b9-847b-cc7caca2250d-kube-api-access-f7688\") pod \"maas-api-revoke-sub-2c1064da-zvcw6\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.327368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.327220    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c1064da-zvcw6\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.327368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.327264    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c1064da-zvcw6\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.327956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.327921    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c1064da-zvcw6\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.329966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.329946    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c1064da-zvcw6\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.338793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.338770    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7688\" (UniqueName: \"kubernetes.io/projected/4e96f89d-f177-43b9-847b-cc7caca2250d-kube-api-access-f7688\") pod \"maas-api-revoke-sub-2c1064da-zvcw6\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.381658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.381625    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:29.672470 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:22:29.672429    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e96f89d_f177_43b9_847b_cc7caca2250d.slice/crio-b3dc39322063694f2e89f2d59166b48abfa1bb3ea105af6a2eb6a8a12c9ac6bb WatchSource:0}: Error finding container b3dc39322063694f2e89f2d59166b48abfa1bb3ea105af6a2eb6a8a12c9ac6bb: Status 404 returned error can't find the container with id b3dc39322063694f2e89f2d59166b48abfa1bb3ea105af6a2eb6a8a12c9ac6bb
Oct 30 19:22:29.672792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.672763    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"]
Oct 30 19:22:29.767142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.767104    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:29.800719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:29.798669    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6" event={"ID":"4e96f89d-f177-43b9-847b-cc7caca2250d","Type":"ContainerStarted","Data":"b3dc39322063694f2e89f2d59166b48abfa1bb3ea105af6a2eb6a8a12c9ac6bb"}
Oct 30 19:22:30.643776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:30.643746    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:30.804579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:30.804547    4925 generic.go:356] "Generic (PLEG): container finished" podID="4e96f89d-f177-43b9-847b-cc7caca2250d" containerID="a3aa9d40ede6336ffce84fdaa6a0b7f90e190e019e44dc1ed071ed5b77f74cb1" exitCode=0
Oct 30 19:22:30.804762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:30.804608    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6" event={"ID":"4e96f89d-f177-43b9-847b-cc7caca2250d","Type":"ContainerDied","Data":"a3aa9d40ede6336ffce84fdaa6a0b7f90e190e019e44dc1ed071ed5b77f74cb1"}
Oct 30 19:22:31.282043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:31.282009    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:31.949879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:31.949856    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:32.010790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.010739    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:32.061214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.061126    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-xks-ca\") pod \"4e96f89d-f177-43b9-847b-cc7caca2250d\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") "
Oct 30 19:22:32.061562 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.061537    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-openshift-ca\") pod \"4e96f89d-f177-43b9-847b-cc7caca2250d\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") "
Oct 30 19:22:32.061903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.061867    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-openshift-ca" pod "4e96f89d-f177-43b9-847b-cc7caca2250d" (UID: "4e96f89d-f177-43b9-847b-cc7caca2250d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:32.062063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.061928    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e96f89d-f177-43b9-847b-cc7caca2250d-kube-api-access-f7688\" (UniqueName: \"kubernetes.io/projected/4e96f89d-f177-43b9-847b-cc7caca2250d-kube-api-access-f7688\") pod \"4e96f89d-f177-43b9-847b-cc7caca2250d\" (UID: \"4e96f89d-f177-43b9-847b-cc7caca2250d\") "
Oct 30 19:22:32.062436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.062415    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:32.064521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.064482    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-xks-ca" pod "4e96f89d-f177-43b9-847b-cc7caca2250d" (UID: "4e96f89d-f177-43b9-847b-cc7caca2250d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:32.072579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.072529    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e96f89d-f177-43b9-847b-cc7caca2250d-kube-api-access-f7688" pod "4e96f89d-f177-43b9-847b-cc7caca2250d" (UID: "4e96f89d-f177-43b9-847b-cc7caca2250d"). InnerVolumeSpecName "kube-api-access-f7688". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:32.163308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.163280    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7688\" (UniqueName: \"kubernetes.io/projected/4e96f89d-f177-43b9-847b-cc7caca2250d-kube-api-access-f7688\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:32.163308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.163307    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e96f89d-f177-43b9-847b-cc7caca2250d-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:32.741849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.741807    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:32.817604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.817568    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6" event={"ID":"4e96f89d-f177-43b9-847b-cc7caca2250d","Type":"ContainerDied","Data":"b3dc39322063694f2e89f2d59166b48abfa1bb3ea105af6a2eb6a8a12c9ac6bb"}
Oct 30 19:22:32.817604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.817603    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"
Oct 30 19:22:32.817810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:32.817620    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3dc39322063694f2e89f2d59166b48abfa1bb3ea105af6a2eb6a8a12c9ac6bb"
Oct 30 19:22:33.431528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:33.431491    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:34.397919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.397878    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:34.808528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.808439    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"]
Oct 30 19:22:34.815497 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.815471    4925 cpu_manager.go:414] "containerMap: removing container" podUID="4e96f89d-f177-43b9-847b-cc7caca2250d" containerName="revoke-keys"
Oct 30 19:22:34.815720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.815698    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e96f89d-f177-43b9-847b-cc7caca2250d" containerName="revoke-keys"
Oct 30 19:22:34.824211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.816132    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e96f89d-f177-43b9-847b-cc7caca2250d" containerName="revoke-keys"
Oct 30 19:22:34.835884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.835861    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:34.837234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.837215    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"]
Oct 30 19:22:34.848407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.848350    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39b867\"/\"distinct-2-w2-39b867-kserve-self-signed-certs\""
Oct 30 19:22:34.909712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.909460    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-model-cache\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:34.909712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.909533    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-home\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:34.909712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.909573    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7vvk\" (UniqueName: \"kubernetes.io/projected/c548c971-fe34-451d-a52d-73bbb9326e57-kube-api-access-p7vvk\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:34.910046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.909730    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-dshm\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:34.910046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.909819    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-tmp-dir\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:34.910046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.909849    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c548c971-fe34-451d-a52d-73bbb9326e57-tls-certs\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:34.927254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:34.927225    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:35.013849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.013816    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-dshm\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.015382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.015353    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-tmp-dir\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.015951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.015930    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c548c971-fe34-451d-a52d-73bbb9326e57-tls-certs\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.016261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.016246    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-model-cache\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.016438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.016421    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-home\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.016870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.016850    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7vvk\" (UniqueName: \"kubernetes.io/projected/c548c971-fe34-451d-a52d-73bbb9326e57-kube-api-access-p7vvk\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.017155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.015871    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-tmp-dir\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.017282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.016892    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-home\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.017382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.016792    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-model-cache\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.039711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.036009    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-dshm\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.050712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.050626    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c548c971-fe34-451d-a52d-73bbb9326e57-tls-certs\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.053485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.053397    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7vvk\" (UniqueName: \"kubernetes.io/projected/c548c971-fe34-451d-a52d-73bbb9326e57-kube-api-access-p7vvk\") pod \"distinct-2-w2-39b867-kserve-576dc667d5-fg6ft\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") " pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.161351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.156909    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:35.455795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.452203    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:35.538713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.537590    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"]
Oct 30 19:22:35.898156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.898127    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" event={"ID":"c548c971-fe34-451d-a52d-73bbb9326e57","Type":"ContainerStarted","Data":"9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698"}
Oct 30 19:22:35.898542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.898167    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" event={"ID":"c548c971-fe34-451d-a52d-73bbb9326e57","Type":"ContainerStarted","Data":"b6d7430a648883bc2b3f1c325a84e86576734feeb032e4dac8f7a18c353452df"}
Oct 30 19:22:35.915280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:35.915200    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podStartSLOduration=1.915183469 podStartE2EDuration="1.915183469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:35.91478538 +0000 UTC m=+990.817798417" watchObservedRunningTime="2026-10-30 19:22:35.915183469 +0000 UTC m=+990.818196508"
Oct 30 19:22:36.156817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:36.156624    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:36.158140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:36.158117    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:36.943914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:36.943875    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:37.543852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:37.543802    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:38.351209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:38.351171    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:39.145634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:39.145600    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:39.801818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:39.801783    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:40.490259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:40.490222    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:41.430415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:41.430382    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:42.052486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:42.052453    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:42.775418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:42.775358    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:43.696842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:43.696807    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:44.468716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:44.467477    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:45.020827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.020783    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.070837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.070796    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.092761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.091779    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.138842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.137789    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.143996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.143961    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.153055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.153012    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.162300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.162271    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:45.214723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.214662    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:45.268113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.268077    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.340988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.340858    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:22:45.809714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.799323    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:45.919896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:45.919858    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Oct 30 19:22:46.182731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:46.182450    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:22:46.446201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:46.446115    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:47.296392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:47.296362    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:47.970142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:47.970078    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:48.590309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:48.590263    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:48.892732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:48.892702    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"]
Oct 30 19:22:48.897144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:48.897129    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:48.901413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:48.901384    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:48.901643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:48.901566    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:22:48.906223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:48.906202    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"]
Oct 30 19:22:48.991091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:48.991058    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81cc5519-q6k5z\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:48.991292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:48.991121    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81cc5519-q6k5z\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:48.991292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:48.991229    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcns5\" (UniqueName: \"kubernetes.io/projected/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-kube-api-access-fcns5\") pod \"maas-api-revoke-sub-81cc5519-q6k5z\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:49.092060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.092027    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcns5\" (UniqueName: \"kubernetes.io/projected/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-kube-api-access-fcns5\") pod \"maas-api-revoke-sub-81cc5519-q6k5z\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:49.092236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.092170    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81cc5519-q6k5z\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:49.092236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.092229    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81cc5519-q6k5z\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:49.092950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.092931    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81cc5519-q6k5z\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:49.094888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.094866    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81cc5519-q6k5z\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:49.103081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.103059    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcns5\" (UniqueName: \"kubernetes.io/projected/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-kube-api-access-fcns5\") pod \"maas-api-revoke-sub-81cc5519-q6k5z\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:49.222324 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.222198    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:49.295206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.295170    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:49.505376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.505340    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"]
Oct 30 19:22:49.506751 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:22:49.506718    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5288c3d_7d94_4ab4_8bb5_2d0c162b80c6.slice/crio-605810a5127a7edb8179297cf875e0718dca95473c7a7869b55c1ff53888ccba WatchSource:0}: Error finding container 605810a5127a7edb8179297cf875e0718dca95473c7a7869b55c1ff53888ccba: Status 404 returned error can't find the container with id 605810a5127a7edb8179297cf875e0718dca95473c7a7869b55c1ff53888ccba
Oct 30 19:22:49.848752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:49.848713    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:50.112240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.112167    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z" event={"ID":"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6","Type":"ContainerStarted","Data":"b9012b46285a6b39c79b2ebef9e01129ac70627389b9cf574e3a9b56f6a7f7ed"}
Oct 30 19:22:50.112240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.112218    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z" event={"ID":"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6","Type":"ContainerStarted","Data":"605810a5127a7edb8179297cf875e0718dca95473c7a7869b55c1ff53888ccba"}
Oct 30 19:22:50.133256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.133205    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z" podStartSLOduration=2.133189215 podStartE2EDuration="2.133189215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:50.130102398 +0000 UTC m=+1005.033115435" watchObservedRunningTime="2026-10-30 19:22:50.133189215 +0000 UTC m=+1005.036202253"
Oct 30 19:22:50.155089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.155058    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"]
Oct 30 19:22:50.159983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.159962    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.174499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.174472    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"]
Oct 30 19:22:50.212207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.212175    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.212346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.212225    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qkt\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-kube-api-access-z5qkt\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.212346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.212259    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.212489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.212464    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.212548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.212535    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.212585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.212568    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.212632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.212599    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.212779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.212758    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.212896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.212879    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.313758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.313723    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.313899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.313779    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qkt\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-kube-api-access-z5qkt\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.313962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.313909    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.314022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314005    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.314079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314049    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.314135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314092    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.314175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314135    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.314220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314196    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.314264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314222    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.314659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314610    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.314809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314673    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.314932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314906    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.315063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.314982    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.315063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.315004    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.316722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.316674    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.316899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.316883    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.328121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.328097    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.328368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.328261    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qkt\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-kube-api-access-z5qkt\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.378056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.377993    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"]
Oct 30 19:22:50.397086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.397062    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.411700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.411661    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"]
Oct 30 19:22:50.477548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.477499    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:50.502822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.502791    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"]
Oct 30 19:22:50.507851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.507830    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.518321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.518284    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.518449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.518349    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.518517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.518461    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.518577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.518513    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.518653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.518634    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.518794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.518771    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9215648-6e52-454f-b9cd-1e2985794e9f-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.518862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.518840    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.518991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.518972    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.519052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.519007    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9hq\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-kube-api-access-cb9hq\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.577024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.576992    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"]
Oct 30 19:22:50.623700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.623098    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.623700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.623160    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.623700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.623213    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.623700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.623330    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.623700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.623382    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.623700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.623484    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.623700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.623624    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.624333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.623730    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.624333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.623816    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9215648-6e52-454f-b9cd-1e2985794e9f-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.624333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624033    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.624333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624085    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.624333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624277    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.624333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624309    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9hq\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-kube-api-access-cb9hq\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.624585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624367    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.624585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624389    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.624585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624433    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.624585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624472    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.624585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624544    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtzf\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-kube-api-access-2dtzf\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.624823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624601    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.624823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624618    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9215648-6e52-454f-b9cd-1e2985794e9f-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.624926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624896    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.624988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.624974    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.625715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.625675    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.627237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.627210    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.627819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.627748    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.635422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.635372    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.636244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.636221    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9hq\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-kube-api-access-cb9hq\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.726329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726299    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726391    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726465    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726555    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726580    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726617    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726654    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtzf\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-kube-api-access-2dtzf\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726730    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726815    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726827    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-credential-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.726911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.726888    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-data\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.727122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.727101    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istiod-ca-cert\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.727208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.727135    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-certs\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.727208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.727167    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-socket\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.728957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.728934    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-envoy\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.729272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.729252    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-podinfo\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.738250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.738229    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-token\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.738492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.738472    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtzf\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-kube-api-access-2dtzf\") pod \"e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") " pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.762278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.762258    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:50.769722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.769665    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"]
Oct 30 19:22:50.773841 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:22:50.773821    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b37d28_6688_46cb_b2e4_b1bc7495eaa0.slice/crio-c5d435cd90dca621f42a015e5f32f339df7f3bad8d89fe8f06003377e16c3c21 WatchSource:0}: Error finding container c5d435cd90dca621f42a015e5f32f339df7f3bad8d89fe8f06003377e16c3c21: Status 404 returned error can't find the container with id c5d435cd90dca621f42a015e5f32f339df7f3bad8d89fe8f06003377e16c3c21
Oct 30 19:22:50.777015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.776978    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:50.777155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.777057    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:50.777155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.777101    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:50.828300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.828096    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:50.940128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:50.940099    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:51.060901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.060876    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"]
Oct 30 19:22:51.067340 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:22:51.067309    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9215648_6e52_454f_b9cd_1e2985794e9f.slice/crio-7982bcc6e0d602d0800b64f27bd5fc6741a65b05617227c54582bd87cbe2dfa5 WatchSource:0}: Error finding container 7982bcc6e0d602d0800b64f27bd5fc6741a65b05617227c54582bd87cbe2dfa5: Status 404 returned error can't find the container with id 7982bcc6e0d602d0800b64f27bd5fc6741a65b05617227c54582bd87cbe2dfa5
Oct 30 19:22:51.073503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.073463    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:51.073628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.073584    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:51.073722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.073635    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:51.120571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.120540    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" event={"ID":"56b37d28-6688-46cb-b2e4-b1bc7495eaa0","Type":"ContainerStarted","Data":"e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f"}
Oct 30 19:22:51.120754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.120601    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" event={"ID":"56b37d28-6688-46cb-b2e4-b1bc7495eaa0","Type":"ContainerStarted","Data":"c5d435cd90dca621f42a015e5f32f339df7f3bad8d89fe8f06003377e16c3c21"}
Oct 30 19:22:51.122423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.122389    4925 generic.go:356] "Generic (PLEG): container finished" podID="e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6" containerID="b9012b46285a6b39c79b2ebef9e01129ac70627389b9cf574e3a9b56f6a7f7ed" exitCode=0
Oct 30 19:22:51.122559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.122484    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z" event={"ID":"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6","Type":"ContainerDied","Data":"b9012b46285a6b39c79b2ebef9e01129ac70627389b9cf574e3a9b56f6a7f7ed"}
Oct 30 19:22:51.123977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.123953    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" event={"ID":"d9215648-6e52-454f-b9cd-1e2985794e9f","Type":"ContainerStarted","Data":"7982bcc6e0d602d0800b64f27bd5fc6741a65b05617227c54582bd87cbe2dfa5"}
Oct 30 19:22:51.147817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.147771    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podStartSLOduration=1.147758195 podStartE2EDuration="1.147758195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:51.145444572 +0000 UTC m=+1006.048457610" watchObservedRunningTime="2026-10-30 19:22:51.147758195 +0000 UTC m=+1006.050771232"
Oct 30 19:22:51.163511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.163488    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"]
Oct 30 19:22:51.170887 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:22:51.170857    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8cb6ca_1b2a_4cef_95b4_c13f7eba7174.slice/crio-83fa5caac0c5f53e3f96d0a4dd3929ca199e20d617e8679b67a0aa7365e6233f WatchSource:0}: Error finding container 83fa5caac0c5f53e3f96d0a4dd3929ca199e20d617e8679b67a0aa7365e6233f: Status 404 returned error can't find the container with id 83fa5caac0c5f53e3f96d0a4dd3929ca199e20d617e8679b67a0aa7365e6233f
Oct 30 19:22:51.178485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.178449    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:51.178586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.178526    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:51.178586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.178573    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:22:51.478520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.478435    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:51.480119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.480095    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:22:51.480264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.480147    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:22:51.819563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:51.819418    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:52.186432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.186391    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" event={"ID":"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174","Type":"ContainerStarted","Data":"69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234"}
Oct 30 19:22:52.186884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.186466    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" event={"ID":"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174","Type":"ContainerStarted","Data":"83fa5caac0c5f53e3f96d0a4dd3929ca199e20d617e8679b67a0aa7365e6233f"}
Oct 30 19:22:52.190234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.190207    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" event={"ID":"d9215648-6e52-454f-b9cd-1e2985794e9f","Type":"ContainerStarted","Data":"7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c"}
Oct 30 19:22:52.256730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.241318    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" podStartSLOduration=2.241298595 podStartE2EDuration="2.241298595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:52.225057339 +0000 UTC m=+1007.128070381" watchObservedRunningTime="2026-10-30 19:22:52.241298595 +0000 UTC m=+1007.144311633"
Oct 30 19:22:52.287407 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:22:52.275002    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:22:52.287407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.283862    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" podStartSLOduration=2.283841654 podStartE2EDuration="2.283841654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:52.268923769 +0000 UTC m=+1007.171936806" watchObservedRunningTime="2026-10-30 19:22:52.283841654 +0000 UTC m=+1007.186854692"
Oct 30 19:22:52.480092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.479630    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:22:52.480092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.479708    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:22:52.774779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.774740    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:52.785758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.785399    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:22:52.785758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.785452    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:22:52.793121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.793098    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:52.843137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.842746    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:52.843137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.842869    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:22:52.843137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.842912    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:22:52.891510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.891470    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-xks-ca\") pod \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") "
Oct 30 19:22:52.891762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.891602    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-openshift-ca\") pod \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") "
Oct 30 19:22:52.891844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.891772    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-kube-api-access-fcns5\" (UniqueName: \"kubernetes.io/projected/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-kube-api-access-fcns5\") pod \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\" (UID: \"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6\") "
Oct 30 19:22:52.893121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.893095    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-openshift-ca" pod "e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6" (UID: "e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:52.953732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.949868    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-kube-api-access-fcns5" pod "e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6" (UID: "e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6"). InnerVolumeSpecName "kube-api-access-fcns5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:52.978722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.973831    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-xks-ca" pod "e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6" (UID: "e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:52.997792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.994807    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:52.997792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.994845    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:52.997792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:52.994864    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcns5\" (UniqueName: \"kubernetes.io/projected/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6-kube-api-access-fcns5\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:53.085734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.078157    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:53.494789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.494751    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:22:53.495391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.495360    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:22:53.579096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.579052    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"]
Oct 30 19:22:53.579878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.579857    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6" containerName="revoke-keys"
Oct 30 19:22:53.579878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.579879    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6" containerName="revoke-keys"
Oct 30 19:22:53.580073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.580011    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6" containerName="revoke-keys"
Oct 30 19:22:53.588057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.588040    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"]
Oct 30 19:22:53.588208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.588143    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.590406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.590379    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z" event={"ID":"e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6","Type":"ContainerDied","Data":"605810a5127a7edb8179297cf875e0718dca95473c7a7869b55c1ff53888ccba"}
Oct 30 19:22:53.590509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.590449    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="605810a5127a7edb8179297cf875e0718dca95473c7a7869b55c1ff53888ccba"
Oct 30 19:22:53.590573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.590540    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"
Oct 30 19:22:53.690992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.690957    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcbd\" (UniqueName: \"kubernetes.io/projected/8830d16d-a8d9-44ac-be97-c98a0e86f421-kube-api-access-cgcbd\") pod \"maas-api-revoke-sub-79cf21af-h6sg4\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.691173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.691021    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cf21af-h6sg4\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.691173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.691089    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cf21af-h6sg4\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.764772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.764090    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:22:53.764772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.764156    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:22:53.792916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.792885    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcbd\" (UniqueName: \"kubernetes.io/projected/8830d16d-a8d9-44ac-be97-c98a0e86f421-kube-api-access-cgcbd\") pod \"maas-api-revoke-sub-79cf21af-h6sg4\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.793091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.792926    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cf21af-h6sg4\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.793091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.793005    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cf21af-h6sg4\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.793843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.793818    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79cf21af-h6sg4\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.815714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.815320    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcbd\" (UniqueName: \"kubernetes.io/projected/8830d16d-a8d9-44ac-be97-c98a0e86f421-kube-api-access-cgcbd\") pod \"maas-api-revoke-sub-79cf21af-h6sg4\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.827716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.826318    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79cf21af-h6sg4\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:53.831710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.830155    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:22:53.831710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.830210    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:22:53.972715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:53.971984    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:54.178711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:54.155028    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:54.488266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:54.487774    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:22:54.488266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:54.487833    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:22:54.780978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:54.767063    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:22:54.780978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:54.767122    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:22:54.874382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:54.873927    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:22:54.874382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:54.873987    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:22:54.981751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:54.979508    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"]
Oct 30 19:22:55.326831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.312569    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:55.480861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.480797    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:22:55.481584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.481551    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:22:55.787350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.787305    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4" event={"ID":"8830d16d-a8d9-44ac-be97-c98a0e86f421","Type":"ContainerStarted","Data":"536e05a56d5cfc76187a44ac54d238dc6c78dc8f0885ae8da029281ab3794dd8"}
Oct 30 19:22:55.789940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.789909    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4" event={"ID":"8830d16d-a8d9-44ac-be97-c98a0e86f421","Type":"ContainerStarted","Data":"74c006d286778370420c809383ff76a3bcde7a65b0d5be2f39fb14d56827bb38"}
Oct 30 19:22:55.790119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.788364    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:22:55.790270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.790250    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:22:55.877782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.876196    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:22:55.877782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.876261    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:22:55.887715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:55.879762    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4" podStartSLOduration=2.879744722 podStartE2EDuration="2.879744722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:55.878567987 +0000 UTC m=+1010.781581025" watchObservedRunningTime="2026-10-30 19:22:55.879744722 +0000 UTC m=+1010.782757760"
Oct 30 19:22:56.108716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.106656    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:56.478415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.478058    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:22:56.478415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.478113    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:22:56.543454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.542374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:56.766774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.765834    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:22:56.766774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.765892    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:22:56.834716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.834095    4925 generic.go:356] "Generic (PLEG): container finished" podID="8830d16d-a8d9-44ac-be97-c98a0e86f421" containerID="536e05a56d5cfc76187a44ac54d238dc6c78dc8f0885ae8da029281ab3794dd8" exitCode=0
Oct 30 19:22:56.834716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.834313    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4" event={"ID":"8830d16d-a8d9-44ac-be97-c98a0e86f421","Type":"ContainerDied","Data":"536e05a56d5cfc76187a44ac54d238dc6c78dc8f0885ae8da029281ab3794dd8"}
Oct 30 19:22:56.836228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.836205    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:22:56.836412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:56.836390    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:22:57.478742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:57.478704    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:22:57.478899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:57.478758    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:22:57.762777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:57.762732    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:22:57.762996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:57.762810    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:22:57.828786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:57.828745    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:22:57.828941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:57.828810    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:22:58.051852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.051819    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:58.095325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.095128    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:58.154670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.154239    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8830d16d-a8d9-44ac-be97-c98a0e86f421-kube-api-access-cgcbd\" (UniqueName: \"kubernetes.io/projected/8830d16d-a8d9-44ac-be97-c98a0e86f421-kube-api-access-cgcbd\") pod \"8830d16d-a8d9-44ac-be97-c98a0e86f421\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") "
Oct 30 19:22:58.154670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.154390    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-xks-ca\") pod \"8830d16d-a8d9-44ac-be97-c98a0e86f421\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") "
Oct 30 19:22:58.154670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.154485    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-openshift-ca\") pod \"8830d16d-a8d9-44ac-be97-c98a0e86f421\" (UID: \"8830d16d-a8d9-44ac-be97-c98a0e86f421\") "
Oct 30 19:22:58.161295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.161263    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-openshift-ca" pod "8830d16d-a8d9-44ac-be97-c98a0e86f421" (UID: "8830d16d-a8d9-44ac-be97-c98a0e86f421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:58.161580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.161561    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8830d16d-a8d9-44ac-be97-c98a0e86f421-kube-api-access-cgcbd" pod "8830d16d-a8d9-44ac-be97-c98a0e86f421" (UID: "8830d16d-a8d9-44ac-be97-c98a0e86f421"). InnerVolumeSpecName "kube-api-access-cgcbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:58.182182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.182143    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-xks-ca" pod "8830d16d-a8d9-44ac-be97-c98a0e86f421" (UID: "8830d16d-a8d9-44ac-be97-c98a0e86f421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:58.256566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.256531    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:58.256566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.256570    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8830d16d-a8d9-44ac-be97-c98a0e86f421-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:58.256803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.256587    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgcbd\" (UniqueName: \"kubernetes.io/projected/8830d16d-a8d9-44ac-be97-c98a0e86f421-kube-api-access-cgcbd\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:58.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.478667    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:22:58.478897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.478743    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:22:58.931709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.930309    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4" event={"ID":"8830d16d-a8d9-44ac-be97-c98a0e86f421","Type":"ContainerDied","Data":"74c006d286778370420c809383ff76a3bcde7a65b0d5be2f39fb14d56827bb38"}
Oct 30 19:22:58.931709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.930430    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c006d286778370420c809383ff76a3bcde7a65b0d5be2f39fb14d56827bb38"
Oct 30 19:22:58.931709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.930534    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"
Oct 30 19:22:58.949759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:58.949494    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:59.427724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:59.425023    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:59.427724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:59.426412    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:59.427724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:59.427281    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:22:59.486806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:59.486182    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:59.489335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:59.487282    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:59.489335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:59.488321    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:22:59.713208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:59.706845    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:59.722231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:59.719600    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:22:59.722231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:22:59.721666    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:23:00.581115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:00.579264    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:01.030149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:01.029192    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:01.091715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:01.087894    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:01.133197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:01.133153    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:01.153038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:01.151278    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:01.258395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:01.257822    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:01.268526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:01.268490    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:23:01.343945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:01.343833    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:01.794719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:01.792800    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:02.025840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.025788    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:02.027576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.027539    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:23:02.125797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.114284    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:02.125797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.115138    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:23:02.286714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.260825    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:23:02.453178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.367588    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:02.463786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.453794    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:23:02.463786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.463392    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:23:02.486788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.486741    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:02.717220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.717126    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:23:02.739265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.739235    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:23:02.771489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.771391    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:23:02.853789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:02.853753    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:03.787553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:03.787521    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:04.624406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.624360    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:04.834788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.834753    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"]
Oct 30 19:23:04.835514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.835486    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8830d16d-a8d9-44ac-be97-c98a0e86f421" containerName="revoke-keys"
Oct 30 19:23:04.835514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.835513    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8830d16d-a8d9-44ac-be97-c98a0e86f421" containerName="revoke-keys"
Oct 30 19:23:04.835716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.835625    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8830d16d-a8d9-44ac-be97-c98a0e86f421" containerName="revoke-keys"
Oct 30 19:23:04.847742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.847678    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:04.851964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.851932    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"]
Oct 30 19:23:04.954273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.954241    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:04.954427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.954293    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:04.954427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.954367    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:04.954427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.954407    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:04.954603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.954587    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85dn\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-kube-api-access-n85dn\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:04.954642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.954629    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:04.954741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.954726    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:04.954782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.954768    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:04.954825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:04.954790    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.056489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.056449    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.056489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.056488    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.056774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.056540    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.056774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.056565    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.056774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.056581    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.056774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.056604    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.056973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.056808    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85dn\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-kube-api-access-n85dn\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.056973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.056862    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.056973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.056891    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.057133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.057026    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.057133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.057086    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-certs\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.057230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.057178    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-credential-socket\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.057230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.057193    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-data\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.057365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.057347    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istiod-ca-cert\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.059377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.059357    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-envoy\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.059602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.059582    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-podinfo\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.068743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.068719    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-token\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.069003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.068983    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85dn\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-kube-api-access-n85dn\") pod \"e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") " pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.174122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.174095    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:05.326772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.322892    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"]
Oct 30 19:23:05.329075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.329052    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.353217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.353180    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"]
Oct 30 19:23:05.366928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.364107    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.366928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.364272    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.366928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.364358    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d234af34-05fd-4440-9419-2a32c3bcef37-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.366928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.364473    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttspj\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-kube-api-access-ttspj\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.366928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.364609    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.366928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.364638    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d234af34-05fd-4440-9419-2a32c3bcef37-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.366928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.364744    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.366928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.364779    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.366928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.364805    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467261    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467389    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d234af34-05fd-4440-9419-2a32c3bcef37-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467429    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttspj\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-kube-api-access-ttspj\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467553    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467576    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d234af34-05fd-4440-9419-2a32c3bcef37-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467641    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467669    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467711    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467770    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-certs\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.467848    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.468557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.468049    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-data\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.469348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.468561    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-credential-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.469348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.469054    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-socket\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.470298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.469674    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d234af34-05fd-4440-9419-2a32c3bcef37-istiod-ca-cert\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.471093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.471060    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d234af34-05fd-4440-9419-2a32c3bcef37-istio-podinfo\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.472865 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:05.472842    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb4d366_46d4_4a62_9cb0_733a58254f2f.slice/crio-3442e7955b129d20f6d6ecc21ef1e956ed3097cece24bd779c122e45b88bb956 WatchSource:0}: Error finding container 3442e7955b129d20f6d6ecc21ef1e956ed3097cece24bd779c122e45b88bb956: Status 404 returned error can't find the container with id 3442e7955b129d20f6d6ecc21ef1e956ed3097cece24bd779c122e45b88bb956
Oct 30 19:23:05.475270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.475248    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"]
Oct 30 19:23:05.476092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.476055    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:05.476168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.476136    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:05.476203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.476179    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:05.476802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.476782    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-envoy\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.480654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.480625    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttspj\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-kube-api-access-ttspj\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.480764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.480648    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-istio-token\") pod \"e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") " pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.518875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.518845    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:05.549395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.549353    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" event={"ID":"bcb4d366-46d4-4a62-9cb0-733a58254f2f","Type":"ContainerStarted","Data":"3442e7955b129d20f6d6ecc21ef1e956ed3097cece24bd779c122e45b88bb956"}
Oct 30 19:23:05.659149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.659104    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:05.959983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.959959    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"]
Oct 30 19:23:05.961032 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:05.961008    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd234af34_05fd_4440_9419_2a32c3bcef37.slice/crio-faf81f1a909168daed3874cead20458bc958922ef9840daee2c1b99cbb727e5b WatchSource:0}: Error finding container faf81f1a909168daed3874cead20458bc958922ef9840daee2c1b99cbb727e5b: Status 404 returned error can't find the container with id faf81f1a909168daed3874cead20458bc958922ef9840daee2c1b99cbb727e5b
Oct 30 19:23:05.963545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.963511    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:05.963620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.963580    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:05.963620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:05.963607    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:06.557323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.557286    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" event={"ID":"bcb4d366-46d4-4a62-9cb0-733a58254f2f","Type":"ContainerStarted","Data":"4f030805bd914faffacb27a3183090391f08a9360cb1085551bbb0c5ff2be747"}
Oct 30 19:23:06.559176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.559124    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" event={"ID":"d234af34-05fd-4440-9419-2a32c3bcef37","Type":"ContainerStarted","Data":"dbcb39cdac533e0e8a5cb8a1635f81d612b9f26009a2f797bc34a825b7c3d738"}
Oct 30 19:23:06.559369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.559185    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" event={"ID":"d234af34-05fd-4440-9419-2a32c3bcef37","Type":"ContainerStarted","Data":"faf81f1a909168daed3874cead20458bc958922ef9840daee2c1b99cbb727e5b"}
Oct 30 19:23:06.583665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.583617    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podStartSLOduration=2.583605172 podStartE2EDuration="2.583605172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:06.579784237 +0000 UTC m=+1021.482797274" watchObservedRunningTime="2026-10-30 19:23:06.583605172 +0000 UTC m=+1021.486618279"
Oct 30 19:23:06.621819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.618862    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podStartSLOduration=1.618837495 podStartE2EDuration="1.618837495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:06.608246276 +0000 UTC m=+1021.511259314" watchObservedRunningTime="2026-10-30 19:23:06.618837495 +0000 UTC m=+1021.521850532"
Oct 30 19:23:06.629846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.629651    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:06.633720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.631774    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"]
Oct 30 19:23:06.636372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.636353    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.639667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.639622    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:06.639914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.639895    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:06.644809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.644783    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"]
Oct 30 19:23:06.663734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.659361    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:06.687279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.687203    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b518a7-gvx82\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.687432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.687357    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllrx\" (UniqueName: \"kubernetes.io/projected/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-kube-api-access-bllrx\") pod \"maas-api-revoke-sub-f2b518a7-gvx82\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.687530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.687510    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b518a7-gvx82\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.789081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.789051    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b518a7-gvx82\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.789242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.789203    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b518a7-gvx82\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.789282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.789266    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllrx\" (UniqueName: \"kubernetes.io/projected/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-kube-api-access-bllrx\") pod \"maas-api-revoke-sub-f2b518a7-gvx82\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.789984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.789912    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b518a7-gvx82\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.791951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.791932    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b518a7-gvx82\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.827448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.827357    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllrx\" (UniqueName: \"kubernetes.io/projected/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-kube-api-access-bllrx\") pod \"maas-api-revoke-sub-f2b518a7-gvx82\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:06.965287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:06.965253    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:07.176157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:07.174971    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:07.176157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:07.176118    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:07.176157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:07.176150    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 19:23:07.355013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:07.354978    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"]
Oct 30 19:23:07.566769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:07.566658    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82" event={"ID":"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde","Type":"ContainerStarted","Data":"912b8568e47c96bba5e392b956fcfc70f1eaf76381f728fd37c765257cbc8605"}
Oct 30 19:23:07.661542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:07.661508    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:07.661668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:07.661594    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:23:07.764942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:07.764677    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:08.175045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:08.175014    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:08.175193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:08.175063    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 19:23:08.578925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:08.578888    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82" event={"ID":"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde","Type":"ContainerStarted","Data":"c61f3b0c29511efb750a3fab667a7cd99dd9e1ec96bc77452cc4d6d1e6fa7673"}
Oct 30 19:23:08.650454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:08.650394    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82" podStartSLOduration=2.650373171 podStartE2EDuration="2.650373171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:08.633155988 +0000 UTC m=+1023.536169026" watchObservedRunningTime="2026-10-30 19:23:08.650373171 +0000 UTC m=+1023.553386209"
Oct 30 19:23:08.659596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:08.659564    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:08.659855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:08.659823    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:23:08.948886 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:08.948849    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"]
Oct 30 19:23:08.955469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:08.955445    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:08.956281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:08.956255    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"]
Oct 30 19:23:09.082027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.081970    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af99c774-060e-4e69-913b-4b3657025202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de902364-zgwbm\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.082192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.082178    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvvvn\" (UniqueName: \"kubernetes.io/projected/af99c774-060e-4e69-913b-4b3657025202-kube-api-access-fvvvn\") pod \"maas-api-revoke-sub-de902364-zgwbm\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.082233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.082222    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af99c774-060e-4e69-913b-4b3657025202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de902364-zgwbm\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.175672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.175632    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:09.175842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.175735    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 19:23:09.184026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.183997    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvvn\" (UniqueName: \"kubernetes.io/projected/af99c774-060e-4e69-913b-4b3657025202-kube-api-access-fvvvn\") pod \"maas-api-revoke-sub-de902364-zgwbm\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.184191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.184067    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af99c774-060e-4e69-913b-4b3657025202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de902364-zgwbm\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.184312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.184274    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af99c774-060e-4e69-913b-4b3657025202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de902364-zgwbm\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.185060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.185036    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af99c774-060e-4e69-913b-4b3657025202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de902364-zgwbm\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.187171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.187155    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af99c774-060e-4e69-913b-4b3657025202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de902364-zgwbm\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.215679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.215554    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:09.243670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.243597    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvvn\" (UniqueName: \"kubernetes.io/projected/af99c774-060e-4e69-913b-4b3657025202-kube-api-access-fvvvn\") pod \"maas-api-revoke-sub-de902364-zgwbm\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.273471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.270951    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:09.587939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.587864    4925 generic.go:356] "Generic (PLEG): container finished" podID="737c41c7-66b4-4bbb-a8bd-0bdb2b562bde" containerID="c61f3b0c29511efb750a3fab667a7cd99dd9e1ec96bc77452cc4d6d1e6fa7673" exitCode=0
Oct 30 19:23:09.588176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.587954    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82" event={"ID":"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde","Type":"ContainerDied","Data":"c61f3b0c29511efb750a3fab667a7cd99dd9e1ec96bc77452cc4d6d1e6fa7673"}
Oct 30 19:23:09.652988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.652361    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"]
Oct 30 19:23:09.655110 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:09.655085    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf99c774_060e_4e69_913b_4b3657025202.slice/crio-b1d8f8a9f8b1bcbf7b9e9bbaec8c9c3d4aebb951aa347a28503be45cba53401d WatchSource:0}: Error finding container b1d8f8a9f8b1bcbf7b9e9bbaec8c9c3d4aebb951aa347a28503be45cba53401d: Status 404 returned error can't find the container with id b1d8f8a9f8b1bcbf7b9e9bbaec8c9c3d4aebb951aa347a28503be45cba53401d
Oct 30 19:23:09.664190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.662970    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:09.664190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:09.663009    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:23:10.175471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:10.175437    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:10.175862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:10.175483    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 19:23:10.294082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:10.293993    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:10.648540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:10.647770    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm" event={"ID":"af99c774-060e-4e69-913b-4b3657025202","Type":"ContainerStarted","Data":"e659d343dfc752551cbf2dc3f490e2528774507cc719e6b54fbe4637c36a2982"}
Oct 30 19:23:10.648540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:10.647827    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm" event={"ID":"af99c774-060e-4e69-913b-4b3657025202","Type":"ContainerStarted","Data":"b1d8f8a9f8b1bcbf7b9e9bbaec8c9c3d4aebb951aa347a28503be45cba53401d"}
Oct 30 19:23:10.660475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:10.660446    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:10.660656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:10.660637    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:23:10.666380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:10.666337    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm" podStartSLOduration=2.666323641 podStartE2EDuration="2.666323641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:10.663796318 +0000 UTC m=+1025.566809355" watchObservedRunningTime="2026-10-30 19:23:10.666323641 +0000 UTC m=+1025.569336702"
Oct 30 19:23:11.050849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.050796    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:11.162311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.162216    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-openshift-ca\") pod \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") "
Oct 30 19:23:11.162522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.162365    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-xks-ca\") pod \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") "
Oct 30 19:23:11.162588 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.162542    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-kube-api-access-bllrx\" (UniqueName: \"kubernetes.io/projected/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-kube-api-access-bllrx\") pod \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\" (UID: \"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde\") "
Oct 30 19:23:11.163081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.163024    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-openshift-ca" pod "737c41c7-66b4-4bbb-a8bd-0bdb2b562bde" (UID: "737c41c7-66b4-4bbb-a8bd-0bdb2b562bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:11.165128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.165100    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-xks-ca" pod "737c41c7-66b4-4bbb-a8bd-0bdb2b562bde" (UID: "737c41c7-66b4-4bbb-a8bd-0bdb2b562bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:11.165807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.165777    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-kube-api-access-bllrx" pod "737c41c7-66b4-4bbb-a8bd-0bdb2b562bde" (UID: "737c41c7-66b4-4bbb-a8bd-0bdb2b562bde"). InnerVolumeSpecName "kube-api-access-bllrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:11.175016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.174995    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:11.175131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.175049    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 19:23:11.264478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.264439    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bllrx\" (UniqueName: \"kubernetes.io/projected/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-kube-api-access-bllrx\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:11.264478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.264470    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:11.264478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.264486    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:11.404715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.398217    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:11.655183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.655156    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"
Oct 30 19:23:11.655356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.655183    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82" event={"ID":"737c41c7-66b4-4bbb-a8bd-0bdb2b562bde","Type":"ContainerDied","Data":"912b8568e47c96bba5e392b956fcfc70f1eaf76381f728fd37c765257cbc8605"}
Oct 30 19:23:11.655356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.655239    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="912b8568e47c96bba5e392b956fcfc70f1eaf76381f728fd37c765257cbc8605"
Oct 30 19:23:11.656943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.656914    4925 generic.go:356] "Generic (PLEG): container finished" podID="af99c774-060e-4e69-913b-4b3657025202" containerID="e659d343dfc752551cbf2dc3f490e2528774507cc719e6b54fbe4637c36a2982" exitCode=0
Oct 30 19:23:11.657101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.656970    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm" event={"ID":"af99c774-060e-4e69-913b-4b3657025202","Type":"ContainerDied","Data":"e659d343dfc752551cbf2dc3f490e2528774507cc719e6b54fbe4637c36a2982"}
Oct 30 19:23:11.659599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.659570    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:11.659777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:11.659619    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:23:12.174961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.174924    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:23:12.175134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.174979    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 19:23:12.659845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.659807    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:23:12.660462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.660416    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:23:12.893059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.893022    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:12.901178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.901151    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:12.991154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.991076    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af99c774-060e-4e69-913b-4b3657025202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af99c774-060e-4e69-913b-4b3657025202-maas-api-openshift-ca\") pod \"af99c774-060e-4e69-913b-4b3657025202\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") "
Oct 30 19:23:12.991464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.991443    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af99c774-060e-4e69-913b-4b3657025202-maas-api-openshift-ca" pod "af99c774-060e-4e69-913b-4b3657025202" (UID: "af99c774-060e-4e69-913b-4b3657025202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:12.991643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.991624    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af99c774-060e-4e69-913b-4b3657025202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af99c774-060e-4e69-913b-4b3657025202-maas-api-xks-ca\") pod \"af99c774-060e-4e69-913b-4b3657025202\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") "
Oct 30 19:23:12.991779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.991669    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af99c774-060e-4e69-913b-4b3657025202-kube-api-access-fvvvn\" (UniqueName: \"kubernetes.io/projected/af99c774-060e-4e69-913b-4b3657025202-kube-api-access-fvvvn\") pod \"af99c774-060e-4e69-913b-4b3657025202\" (UID: \"af99c774-060e-4e69-913b-4b3657025202\") "
Oct 30 19:23:12.992217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.992185    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af99c774-060e-4e69-913b-4b3657025202-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:12.998724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.994483    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af99c774-060e-4e69-913b-4b3657025202-maas-api-xks-ca" pod "af99c774-060e-4e69-913b-4b3657025202" (UID: "af99c774-060e-4e69-913b-4b3657025202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:12.998724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:12.994799    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af99c774-060e-4e69-913b-4b3657025202-kube-api-access-fvvvn" pod "af99c774-060e-4e69-913b-4b3657025202" (UID: "af99c774-060e-4e69-913b-4b3657025202"). InnerVolumeSpecName "kube-api-access-fvvvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:13.094100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.094072    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af99c774-060e-4e69-913b-4b3657025202-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:13.094100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.094098    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvvvn\" (UniqueName: \"kubernetes.io/projected/af99c774-060e-4e69-913b-4b3657025202-kube-api-access-fvvvn\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:13.178358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.178332    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:13.178859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.178837    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:13.179464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.179443    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:23:13.677381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.677353    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"
Oct 30 19:23:13.677872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.677434    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm" event={"ID":"af99c774-060e-4e69-913b-4b3657025202","Type":"ContainerDied","Data":"b1d8f8a9f8b1bcbf7b9e9bbaec8c9c3d4aebb951aa347a28503be45cba53401d"}
Oct 30 19:23:13.677872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.677489    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1d8f8a9f8b1bcbf7b9e9bbaec8c9c3d4aebb951aa347a28503be45cba53401d"
Oct 30 19:23:13.779809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.779777    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:13.945050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.944958    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:13.945579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.945556    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:13.946986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:13.946966    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:23:14.614496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.614469    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"]
Oct 30 19:23:14.615058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.615044    4925 cpu_manager.go:414] "containerMap: removing container" podUID="737c41c7-66b4-4bbb-a8bd-0bdb2b562bde" containerName="revoke-keys"
Oct 30 19:23:14.615110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.615060    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737c41c7-66b4-4bbb-a8bd-0bdb2b562bde" containerName="revoke-keys"
Oct 30 19:23:14.615110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.615069    4925 cpu_manager.go:414] "containerMap: removing container" podUID="af99c774-060e-4e69-913b-4b3657025202" containerName="revoke-keys"
Oct 30 19:23:14.615110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.615073    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af99c774-060e-4e69-913b-4b3657025202" containerName="revoke-keys"
Oct 30 19:23:14.615209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.615163    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="737c41c7-66b4-4bbb-a8bd-0bdb2b562bde" containerName="revoke-keys"
Oct 30 19:23:14.615209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.615171    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="af99c774-060e-4e69-913b-4b3657025202" containerName="revoke-keys"
Oct 30 19:23:14.618897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.618880    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.622732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.622657    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:14.622898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.622772    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:14.627897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.627864    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"]
Oct 30 19:23:14.646892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.646858    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:14.718027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.717993    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9hhb\" (UniqueName: \"kubernetes.io/projected/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-kube-api-access-c9hhb\") pod \"maas-api-revoke-sub-5a95ac3a-kpf7w\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.718514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.718224    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a95ac3a-kpf7w\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.718514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.718280    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a95ac3a-kpf7w\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.819772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.819735    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9hhb\" (UniqueName: \"kubernetes.io/projected/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-kube-api-access-c9hhb\") pod \"maas-api-revoke-sub-5a95ac3a-kpf7w\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.819945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.819895    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a95ac3a-kpf7w\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.819945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.819918    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a95ac3a-kpf7w\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.820611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.820585    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a95ac3a-kpf7w\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.822506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.822485    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a95ac3a-kpf7w\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.832191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.832165    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9hhb\" (UniqueName: \"kubernetes.io/projected/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-kube-api-access-c9hhb\") pod \"maas-api-revoke-sub-5a95ac3a-kpf7w\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:14.934633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:14.934607    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:15.219927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:15.219898    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"]
Oct 30 19:23:15.221465 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:15.221438    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f7bc46_c6e2_4011_b6dd_4cd0f4d4c8f7.slice/crio-2196bda74a533780d928d9fe04a66e6cfff2c23f09c4e21298a13576eeb73c95 WatchSource:0}: Error finding container 2196bda74a533780d928d9fe04a66e6cfff2c23f09c4e21298a13576eeb73c95: Status 404 returned error can't find the container with id 2196bda74a533780d928d9fe04a66e6cfff2c23f09c4e21298a13576eeb73c95
Oct 30 19:23:15.558042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:15.558008    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:15.700572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:15.700535    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w" event={"ID":"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7","Type":"ContainerStarted","Data":"14d8e9dcbcea3b6481ef3306997761b4d93d104e6f7c9d7e00975e0f5afc2fad"}
Oct 30 19:23:15.700789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:15.700599    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w" event={"ID":"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7","Type":"ContainerStarted","Data":"2196bda74a533780d928d9fe04a66e6cfff2c23f09c4e21298a13576eeb73c95"}
Oct 30 19:23:15.721434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:15.721386    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w" podStartSLOduration=1.7213745 podStartE2EDuration="1.7213745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:15.717303992 +0000 UTC m=+1030.620317028" watchObservedRunningTime="2026-10-30 19:23:15.7213745 +0000 UTC m=+1030.624387536"
Oct 30 19:23:16.245670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:16.245637    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:16.708339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:16.708305    4925 generic.go:356] "Generic (PLEG): container finished" podID="68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7" containerID="14d8e9dcbcea3b6481ef3306997761b4d93d104e6f7c9d7e00975e0f5afc2fad" exitCode=0
Oct 30 19:23:16.708521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:16.708394    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w" event={"ID":"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7","Type":"ContainerDied","Data":"14d8e9dcbcea3b6481ef3306997761b4d93d104e6f7c9d7e00975e0f5afc2fad"}
Oct 30 19:23:16.881428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:16.881397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:17.883191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:17.883155    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:17.946487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:17.946449    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:17.958827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:17.958802    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-openshift-ca\") pod \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") "
Oct 30 19:23:17.959021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:17.958912    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-kube-api-access-c9hhb\" (UniqueName: \"kubernetes.io/projected/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-kube-api-access-c9hhb\") pod \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") "
Oct 30 19:23:17.959021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:17.958932    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-xks-ca\") pod \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\" (UID: \"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7\") "
Oct 30 19:23:17.959306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:17.959247    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-openshift-ca" pod "68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7" (UID: "68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:17.967769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:17.962316    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:17.969111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:17.969069    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-kube-api-access-c9hhb" pod "68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7" (UID: "68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7"). InnerVolumeSpecName "kube-api-access-c9hhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:17.970464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:17.970436    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-xks-ca" pod "68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7" (UID: "68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:18.063757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:18.063678    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9hhb\" (UniqueName: \"kubernetes.io/projected/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-kube-api-access-c9hhb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:18.063757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:18.063751    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:18.737974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:18.737939    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:18.737974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:18.737949    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w" event={"ID":"68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7","Type":"ContainerDied","Data":"2196bda74a533780d928d9fe04a66e6cfff2c23f09c4e21298a13576eeb73c95"}
Oct 30 19:23:18.738209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:18.738016    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"
Oct 30 19:23:18.738265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:18.738019    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2196bda74a533780d928d9fe04a66e6cfff2c23f09c4e21298a13576eeb73c95"
Oct 30 19:23:19.013174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.013097    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"]
Oct 30 19:23:19.013667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.013654    4925 cpu_manager.go:414] "containerMap: removing container" podUID="68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7" containerName="revoke-keys"
Oct 30 19:23:19.013667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.013667    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7" containerName="revoke-keys"
Oct 30 19:23:19.013834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.013822    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7" containerName="revoke-keys"
Oct 30 19:23:19.017551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.017530    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.021238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.021198    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:19.021360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.021304    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:19.029203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.028299    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"]
Oct 30 19:23:19.080020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.079996    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjn2\" (UniqueName: \"kubernetes.io/projected/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-kube-api-access-mvjn2\") pod \"maas-api-revoke-sub-451fba18-hnsnn\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.080154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.080052    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451fba18-hnsnn\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.080313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.080291    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451fba18-hnsnn\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.181649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.181622    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451fba18-hnsnn\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.181808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.181790    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451fba18-hnsnn\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.181930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.181915    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjn2\" (UniqueName: \"kubernetes.io/projected/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-kube-api-access-mvjn2\") pod \"maas-api-revoke-sub-451fba18-hnsnn\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.182354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.182335    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-451fba18-hnsnn\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.184300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.184282    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-451fba18-hnsnn\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.193575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.193549    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjn2\" (UniqueName: \"kubernetes.io/projected/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-kube-api-access-mvjn2\") pod \"maas-api-revoke-sub-451fba18-hnsnn\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.338837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.338763    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:19.543494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.543458    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:19.625680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.625648    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"]
Oct 30 19:23:19.627383 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:19.627290    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bdfd5e_681a_4d20_873d_9b7f8cee77e5.slice/crio-cf54a4aea372e4421228c5f0b058892dfe621a14b4f3c80dc906e7ae9d359adc WatchSource:0}: Error finding container cf54a4aea372e4421228c5f0b058892dfe621a14b4f3c80dc906e7ae9d359adc: Status 404 returned error can't find the container with id cf54a4aea372e4421228c5f0b058892dfe621a14b4f3c80dc906e7ae9d359adc
Oct 30 19:23:19.746787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:19.746753    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn" event={"ID":"53bdfd5e-681a-4d20-873d-9b7f8cee77e5","Type":"ContainerStarted","Data":"cf54a4aea372e4421228c5f0b058892dfe621a14b4f3c80dc906e7ae9d359adc"}
Oct 30 19:23:20.207154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.207082    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"]
Oct 30 19:23:20.211114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.211097    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.222513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.222486    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"]
Oct 30 19:23:20.259672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.259644    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:20.299011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.298975    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.299175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.299042    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.299218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.299174    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.299262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.299242    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a9e3ec-6b63-4906-8647-91384664f2f9-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.299316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.299302    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.299395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.299378    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.299437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.299418    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2vq\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-kube-api-access-rx2vq\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.299486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.299453    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.299602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.299590    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.400988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.400945    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.401358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.401333    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.401707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.401666    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.401832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.401796    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.401894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.401859    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a9e3ec-6b63-4906-8647-91384664f2f9-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.401953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.401911    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.401953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.401947    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.402105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.401979    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2vq\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-kube-api-access-rx2vq\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.402105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.402031    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.402105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.402084    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.402486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.402462    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.402890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.402841    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.403087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.403062    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a9e3ec-6b63-4906-8647-91384664f2f9-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.403170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.403124    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.405515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.405491    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.405766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.405744    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.415899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.415844    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.422213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.422178    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2vq\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-kube-api-access-rx2vq\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.526743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.526715    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:20.755731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.755680    4925 generic.go:356] "Generic (PLEG): container finished" podID="53bdfd5e-681a-4d20-873d-9b7f8cee77e5" containerID="e99f429e99db5050791145deee8280f6621b26c9e200afefc49f5452602d91fc" exitCode=0
Oct 30 19:23:20.755731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.755717    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn" event={"ID":"53bdfd5e-681a-4d20-873d-9b7f8cee77e5","Type":"ContainerDied","Data":"e99f429e99db5050791145deee8280f6621b26c9e200afefc49f5452602d91fc"}
Oct 30 19:23:20.817040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.817016    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"]
Oct 30 19:23:20.817551 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:20.817514    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a9e3ec_6b63_4906_8647_91384664f2f9.slice/crio-2518c37170991f1ae10a0af02b653c0e726da61d893d2e5d751e2c08870c0e85 WatchSource:0}: Error finding container 2518c37170991f1ae10a0af02b653c0e726da61d893d2e5d751e2c08870c0e85: Status 404 returned error can't find the container with id 2518c37170991f1ae10a0af02b653c0e726da61d893d2e5d751e2c08870c0e85
Oct 30 19:23:20.819902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.819871    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:20.819986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.819934    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:20.819986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:20.819961    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:21.247931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:21.247894    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:21.764933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:21.764897    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" event={"ID":"61a9e3ec-6b63-4906-8647-91384664f2f9","Type":"ContainerStarted","Data":"c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b"}
Oct 30 19:23:21.765088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:21.764961    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" event={"ID":"61a9e3ec-6b63-4906-8647-91384664f2f9","Type":"ContainerStarted","Data":"2518c37170991f1ae10a0af02b653c0e726da61d893d2e5d751e2c08870c0e85"}
Oct 30 19:23:21.790770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:21.790711    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podStartSLOduration=1.790675381 podStartE2EDuration="1.790675381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:21.786407423 +0000 UTC m=+1036.689420461" watchObservedRunningTime="2026-10-30 19:23:21.790675381 +0000 UTC m=+1036.693688420"
Oct 30 19:23:21.958519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:21.958498    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:22.023624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.023509    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-openshift-ca\") pod \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") "
Oct 30 19:23:22.024129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.024005    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-openshift-ca" pod "53bdfd5e-681a-4d20-873d-9b7f8cee77e5" (UID: "53bdfd5e-681a-4d20-873d-9b7f8cee77e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:22.024261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.024198    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-kube-api-access-mvjn2\" (UniqueName: \"kubernetes.io/projected/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-kube-api-access-mvjn2\") pod \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") "
Oct 30 19:23:22.024903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.024871    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-xks-ca\") pod \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\" (UID: \"53bdfd5e-681a-4d20-873d-9b7f8cee77e5\") "
Oct 30 19:23:22.025917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.025898    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:22.029109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.027282    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-kube-api-access-mvjn2" pod "53bdfd5e-681a-4d20-873d-9b7f8cee77e5" (UID: "53bdfd5e-681a-4d20-873d-9b7f8cee77e5"). InnerVolumeSpecName "kube-api-access-mvjn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:22.029109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.027392    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-xks-ca" pod "53bdfd5e-681a-4d20-873d-9b7f8cee77e5" (UID: "53bdfd5e-681a-4d20-873d-9b7f8cee77e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:22.080747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.063754    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:22.130262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.130218    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjn2\" (UniqueName: \"kubernetes.io/projected/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-kube-api-access-mvjn2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:22.130427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.130273    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53bdfd5e-681a-4d20-873d-9b7f8cee77e5-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:22.155913 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:23:22.154889    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:23:22.527007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.526972    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:22.528485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.528456    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:22.528621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.528507    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:22.773717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.773660    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"
Oct 30 19:23:22.773889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.773665    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn" event={"ID":"53bdfd5e-681a-4d20-873d-9b7f8cee77e5","Type":"ContainerDied","Data":"cf54a4aea372e4421228c5f0b058892dfe621a14b4f3c80dc906e7ae9d359adc"}
Oct 30 19:23:22.773889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.773815    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf54a4aea372e4421228c5f0b058892dfe621a14b4f3c80dc906e7ae9d359adc"
Oct 30 19:23:22.861931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.861899    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:22.947115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.945492    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"]
Oct 30 19:23:22.947115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.946489    4925 cpu_manager.go:414] "containerMap: removing container" podUID="53bdfd5e-681a-4d20-873d-9b7f8cee77e5" containerName="revoke-keys"
Oct 30 19:23:22.947115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.946509    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bdfd5e-681a-4d20-873d-9b7f8cee77e5" containerName="revoke-keys"
Oct 30 19:23:22.947115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.946724    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bdfd5e-681a-4d20-873d-9b7f8cee77e5" containerName="revoke-keys"
Oct 30 19:23:22.960414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.958773    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"]
Oct 30 19:23:22.960414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.958916    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:22.963949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.962623    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:22.963949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:22.962821    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:23.050414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.050339    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwfk9\" (UniqueName: \"kubernetes.io/projected/82e6cbf8-5117-45b2-9527-28a77a579d00-kube-api-access-bwfk9\") pod \"maas-api-revoke-sub-cce19e59-kc9f5\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.050590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.050550    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce19e59-kc9f5\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.050637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.050613    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce19e59-kc9f5\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.151636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.151606    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce19e59-kc9f5\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.151849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.151674    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce19e59-kc9f5\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.151849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.151792    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwfk9\" (UniqueName: \"kubernetes.io/projected/82e6cbf8-5117-45b2-9527-28a77a579d00-kube-api-access-bwfk9\") pod \"maas-api-revoke-sub-cce19e59-kc9f5\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.152365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.152343    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce19e59-kc9f5\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.154448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.154427    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce19e59-kc9f5\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.163197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.163172    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwfk9\" (UniqueName: \"kubernetes.io/projected/82e6cbf8-5117-45b2-9527-28a77a579d00-kube-api-access-bwfk9\") pod \"maas-api-revoke-sub-cce19e59-kc9f5\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.284273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.284244    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:23.527953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.527922    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:23.528394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.527978    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:23.588075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.588048    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"]
Oct 30 19:23:23.589040 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:23.589015    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e6cbf8_5117_45b2_9527_28a77a579d00.slice/crio-a81c2ee22d676c3fbf620c142b0707fa2a63b219d9d8ba2cdd0a9d020c791935 WatchSource:0}: Error finding container a81c2ee22d676c3fbf620c142b0707fa2a63b219d9d8ba2cdd0a9d020c791935: Status 404 returned error can't find the container with id a81c2ee22d676c3fbf620c142b0707fa2a63b219d9d8ba2cdd0a9d020c791935
Oct 30 19:23:23.761461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.761427    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:23.783170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:23.783142    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5" event={"ID":"82e6cbf8-5117-45b2-9527-28a77a579d00","Type":"ContainerStarted","Data":"a81c2ee22d676c3fbf620c142b0707fa2a63b219d9d8ba2cdd0a9d020c791935"}
Oct 30 19:23:24.539713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:24.532134    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:24.539713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:24.532205    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:24.916732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:24.916610    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5" event={"ID":"82e6cbf8-5117-45b2-9527-28a77a579d00","Type":"ContainerStarted","Data":"2ee3f4866788745a6e3a5ab9534537e690aee5d0981672545198bab08df8921f"}
Oct 30 19:23:24.988714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:24.975544    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5" podStartSLOduration=2.975521928 podStartE2EDuration="2.975521928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:24.971069978 +0000 UTC m=+1039.874083016" watchObservedRunningTime="2026-10-30 19:23:24.975521928 +0000 UTC m=+1039.878534982"
Oct 30 19:23:25.098716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.097780    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:23:25.102796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.102758    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:23:25.103214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.102758    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.155714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.152197    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.155714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.152332    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:23:25.155714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.154474    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.285831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.277342    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.424890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.424792    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.531805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.531753    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:25.532122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.532075    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:25.717208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.717111    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.785512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.785474    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.786236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.785795    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:23:25.918861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.918817    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:25.951805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.951770    4925 generic.go:356] "Generic (PLEG): container finished" podID="82e6cbf8-5117-45b2-9527-28a77a579d00" containerID="2ee3f4866788745a6e3a5ab9534537e690aee5d0981672545198bab08df8921f" exitCode=0
Oct 30 19:23:25.952251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:25.952227    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5" event={"ID":"82e6cbf8-5117-45b2-9527-28a77a579d00","Type":"ContainerDied","Data":"2ee3f4866788745a6e3a5ab9534537e690aee5d0981672545198bab08df8921f"}
Oct 30 19:23:26.118771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.117908    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:23:26.125712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.120835    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.150713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.130089    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:23:26.150713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.130148    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:23:26.175714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.163814    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.175714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.163924    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:23:26.175714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.164490    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:23:26.175714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.164546    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:23:26.175714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.172759    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.278718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.277834    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.278718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.278284    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.278718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.278477    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:23:26.711877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.588976    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.737715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.592215    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:26.793840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.793797    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:26.795323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.721485    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:23:26.841715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.749866    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.841715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.720797    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:23:26.841715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.808466    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:23:26.841715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.795120    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.841715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.810891    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:23:26.875716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.853411    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:26.875716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:26.854170    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:23:27.019960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.004746    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:23:27.146220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.146179    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:23:27.173580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.173537    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 30 19:23:27.174284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.174251    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:23:27.201369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.201309    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.288282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.214900    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:23:27.288635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.288590    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.340731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.246459    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.516811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.477060    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:27.609344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.609253    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:27.632817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.632768    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:27.718963 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.718920    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.797489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.796800    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.829749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.807630    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.829749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.808343    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:23:27.829749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.810461    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:27.856824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.856784    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:23:27.953911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.953874    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:27.954153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:27.954128    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.005627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.005567    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:23:28.006344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.006319    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:23:28.104423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.081458    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.104423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.102870    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.104423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.103591    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:23:28.309812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.157764    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.309812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.185423    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.309812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.299463    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:23:28.309812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.299851    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:23:28.341742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.310218    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:23:28.553874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.539569    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:28.553874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.539621    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:28.564056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.563927    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:28.571725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.567743    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.727853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.727768    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.797058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.797020    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.809833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.809793    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.816168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.815776    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:28.966715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:28.926480    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:23:29.006808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.006768    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:23:29.010223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.010192    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": context deadline exceeded"
Oct 30 19:23:29.083141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.081897    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:23:29.085727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.083781    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:23:29.126311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.122590    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.126311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.123953    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:23:29.191714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.182230    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.191714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.189454    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:29.191714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.189522    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.242341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.218124    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded"
Oct 30 19:23:29.242341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.218342    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:23:29.323511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.323286    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:23:29.400916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.325122    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:29.401353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.401317    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.401919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.392129    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.453725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.451586    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.520725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.509430    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:29.721780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.666513    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:29.721780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.714824    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.721780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.670805    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:29.742851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.740023    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.891549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.891500    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.929715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.927632    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.932023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.931990    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:23:29.962846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.951528    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:29.983535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:29.714929    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:30.016064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.016027    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:23:30.070729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.047581    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5" event={"ID":"82e6cbf8-5117-45b2-9527-28a77a579d00","Type":"ContainerDied","Data":"a81c2ee22d676c3fbf620c142b0707fa2a63b219d9d8ba2cdd0a9d020c791935"}
Oct 30 19:23:30.070729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.047647    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a81c2ee22d676c3fbf620c142b0707fa2a63b219d9d8ba2cdd0a9d020c791935"
Oct 30 19:23:30.084858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.084799    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.156780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.103795    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:30.177801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.170792    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.177801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.155044    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:23:30.177801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.154681    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.180115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.180093    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:23:30.186162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.186133    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:23:30.255744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.255194    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e6cbf8-5117-45b2-9527-28a77a579d00-kube-api-access-bwfk9\" (UniqueName: \"kubernetes.io/projected/82e6cbf8-5117-45b2-9527-28a77a579d00-kube-api-access-bwfk9\") pod \"82e6cbf8-5117-45b2-9527-28a77a579d00\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") "
Oct 30 19:23:30.255744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.255327    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-xks-ca\") pod \"82e6cbf8-5117-45b2-9527-28a77a579d00\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") "
Oct 30 19:23:30.255744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.255387    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-openshift-ca\") pod \"82e6cbf8-5117-45b2-9527-28a77a579d00\" (UID: \"82e6cbf8-5117-45b2-9527-28a77a579d00\") "
Oct 30 19:23:30.264419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.264378    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-openshift-ca" pod "82e6cbf8-5117-45b2-9527-28a77a579d00" (UID: "82e6cbf8-5117-45b2-9527-28a77a579d00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:30.342712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.324797    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-xks-ca" pod "82e6cbf8-5117-45b2-9527-28a77a579d00" (UID: "82e6cbf8-5117-45b2-9527-28a77a579d00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:30.348770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.343037    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.370502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.370462    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:23:30.384214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.384164    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.384587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.384567    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e6cbf8-5117-45b2-9527-28a77a579d00-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.393388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.393337    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.418856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.418817    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:23:30.439232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.439199    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e6cbf8-5117-45b2-9527-28a77a579d00-kube-api-access-bwfk9" pod "82e6cbf8-5117-45b2-9527-28a77a579d00" (UID: "82e6cbf8-5117-45b2-9527-28a77a579d00"). InnerVolumeSpecName "kube-api-access-bwfk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:30.441386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.441361    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:23:30.454678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.454648    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:30.460993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.460973    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:23:30.471710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.471672    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:23:30.476117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.476101    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:23:30.522763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.501766    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwfk9\" (UniqueName: \"kubernetes.io/projected/82e6cbf8-5117-45b2-9527-28a77a579d00-kube-api-access-bwfk9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.564319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.550651    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:23:30.570301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.554439    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:30.570301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.565162    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:30.570301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.569578    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:23:30.580281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.580261    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:23:30.581238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.581211    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:23:30.649620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.649591    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:23:30.672310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:30.672288    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:23:31.032787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:31.032755    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"
Oct 30 19:23:31.178274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:31.177231    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:31.527851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:31.527821    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:23:31.528026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:31.527869    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:23:31.887211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:31.887111    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:32.557320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.556988    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:32.706734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.706700    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"]
Oct 30 19:23:32.707339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.707322    4925 cpu_manager.go:414] "containerMap: removing container" podUID="82e6cbf8-5117-45b2-9527-28a77a579d00" containerName="revoke-keys"
Oct 30 19:23:32.707339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.707341    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e6cbf8-5117-45b2-9527-28a77a579d00" containerName="revoke-keys"
Oct 30 19:23:32.707485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.707475    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e6cbf8-5117-45b2-9527-28a77a579d00" containerName="revoke-keys"
Oct 30 19:23:32.711058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.711042    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:32.715803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.715783    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:32.717982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.717960    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:32.722231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.722090    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"]
Oct 30 19:23:32.747276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.747228    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:32.747880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.747852    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:32.749032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.749005    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:23:32.799726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.798147    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87dc7c05-nw46g\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:32.799726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.798264    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87dc7c05-nw46g\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:32.799726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.798385    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5khq\" (UniqueName: \"kubernetes.io/projected/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-kube-api-access-v5khq\") pod \"maas-api-revoke-sub-87dc7c05-nw46g\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:32.899561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.899517    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87dc7c05-nw46g\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:32.899775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.899589    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87dc7c05-nw46g\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:32.899775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.899659    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5khq\" (UniqueName: \"kubernetes.io/projected/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-kube-api-access-v5khq\") pod \"maas-api-revoke-sub-87dc7c05-nw46g\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:32.900541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.900503    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87dc7c05-nw46g\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:32.902388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.902362    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87dc7c05-nw46g\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:32.920322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:32.920283    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5khq\" (UniqueName: \"kubernetes.io/projected/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-kube-api-access-v5khq\") pod \"maas-api-revoke-sub-87dc7c05-nw46g\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:33.039714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.038102    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:33.344772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.344743    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"]
Oct 30 19:23:33.346975 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:33.346950    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75ddf1c_5514_46f3_83b7_d0c8cf941aa2.slice/crio-605535ef10e1caaef57cb26b8d80e6b58ca912068839eaa294fc0ac68c71d5fb WatchSource:0}: Error finding container 605535ef10e1caaef57cb26b8d80e6b58ca912068839eaa294fc0ac68c71d5fb: Status 404 returned error can't find the container with id 605535ef10e1caaef57cb26b8d80e6b58ca912068839eaa294fc0ac68c71d5fb
Oct 30 19:23:33.459738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.457046    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:33.585743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.585711    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"]
Oct 30 19:23:33.594670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.593318    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.597131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.597105    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"]
Oct 30 19:23:33.727431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.727400    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fcl\" (UniqueName: \"kubernetes.io/projected/f4d8959d-aca3-4c20-8558-3335f9b44729-kube-api-access-62fcl\") pod \"maas-api-revoke-sub-d925576b-xzxs9\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.727681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.727659    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d925576b-xzxs9\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.727828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.727736    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d925576b-xzxs9\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.829247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.829214    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d925576b-xzxs9\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.829393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.829267    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d925576b-xzxs9\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.829393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.829368    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62fcl\" (UniqueName: \"kubernetes.io/projected/f4d8959d-aca3-4c20-8558-3335f9b44729-kube-api-access-62fcl\") pod \"maas-api-revoke-sub-d925576b-xzxs9\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.830383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.830356    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d925576b-xzxs9\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.831927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.831907    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d925576b-xzxs9\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.841737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.841712    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fcl\" (UniqueName: \"kubernetes.io/projected/f4d8959d-aca3-4c20-8558-3335f9b44729-kube-api-access-62fcl\") pod \"maas-api-revoke-sub-d925576b-xzxs9\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:33.936309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:33.936271    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:34.075091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:34.074624    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:34.105733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:34.100040    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g" event={"ID":"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2","Type":"ContainerStarted","Data":"9dc4b6a8acd8d592c9924989214f6136398cc8c7aa5ab7b29f81d2cc97de26a6"}
Oct 30 19:23:34.105733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:34.100097    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g" event={"ID":"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2","Type":"ContainerStarted","Data":"605535ef10e1caaef57cb26b8d80e6b58ca912068839eaa294fc0ac68c71d5fb"}
Oct 30 19:23:34.124856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:34.124796    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g" podStartSLOduration=2.12477629 podStartE2EDuration="2.12477629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:34.124203871 +0000 UTC m=+1049.027216921" watchObservedRunningTime="2026-10-30 19:23:34.12477629 +0000 UTC m=+1049.027789327"
Oct 30 19:23:34.238869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:34.238845    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"]
Oct 30 19:23:34.241099 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:34.241055    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d8959d_aca3_4c20_8558_3335f9b44729.slice/crio-18d3266821faa0523d3a884f087e8c7cfa9639b385393a6d599ac1644f516559 WatchSource:0}: Error finding container 18d3266821faa0523d3a884f087e8c7cfa9639b385393a6d599ac1644f516559: Status 404 returned error can't find the container with id 18d3266821faa0523d3a884f087e8c7cfa9639b385393a6d599ac1644f516559
Oct 30 19:23:34.976449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:34.976413    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:35.106496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:35.106455    4925 generic.go:356] "Generic (PLEG): container finished" podID="b75ddf1c-5514-46f3-83b7-d0c8cf941aa2" containerID="9dc4b6a8acd8d592c9924989214f6136398cc8c7aa5ab7b29f81d2cc97de26a6" exitCode=0
Oct 30 19:23:35.106642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:35.106530    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g" event={"ID":"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2","Type":"ContainerDied","Data":"9dc4b6a8acd8d592c9924989214f6136398cc8c7aa5ab7b29f81d2cc97de26a6"}
Oct 30 19:23:35.108399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:35.108372    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9" event={"ID":"f4d8959d-aca3-4c20-8558-3335f9b44729","Type":"ContainerStarted","Data":"e5fceb67450ffbd358e2641df6f85cd965662968868b7a8287c5c4f566ae3570"}
Oct 30 19:23:35.108505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:35.108422    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9" event={"ID":"f4d8959d-aca3-4c20-8558-3335f9b44729","Type":"ContainerStarted","Data":"18d3266821faa0523d3a884f087e8c7cfa9639b385393a6d599ac1644f516559"}
Oct 30 19:23:35.141669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:35.141624    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9" podStartSLOduration=2.141611553 podStartE2EDuration="2.141611553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:35.139718816 +0000 UTC m=+1050.042731847" watchObservedRunningTime="2026-10-30 19:23:35.141611553 +0000 UTC m=+1050.044624590"
Oct 30 19:23:35.749917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:35.749850    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:36.117485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.117402    4925 generic.go:356] "Generic (PLEG): container finished" podID="f4d8959d-aca3-4c20-8558-3335f9b44729" containerID="e5fceb67450ffbd358e2641df6f85cd965662968868b7a8287c5c4f566ae3570" exitCode=0
Oct 30 19:23:36.117986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.117498    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9" event={"ID":"f4d8959d-aca3-4c20-8558-3335f9b44729","Type":"ContainerDied","Data":"e5fceb67450ffbd358e2641df6f85cd965662968868b7a8287c5c4f566ae3570"}
Oct 30 19:23:36.312777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.312753    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:36.390906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.390726    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:36.414957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.414932    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-xks-ca\") pod \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") "
Oct 30 19:23:36.415172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.415154    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-kube-api-access-v5khq\" (UniqueName: \"kubernetes.io/projected/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-kube-api-access-v5khq\") pod \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") "
Oct 30 19:23:36.415270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.415254    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-openshift-ca\") pod \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\" (UID: \"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2\") "
Oct 30 19:23:36.415600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.415572    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-openshift-ca" pod "b75ddf1c-5514-46f3-83b7-d0c8cf941aa2" (UID: "b75ddf1c-5514-46f3-83b7-d0c8cf941aa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:36.416007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.415985    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:36.417188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.417171    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-xks-ca" pod "b75ddf1c-5514-46f3-83b7-d0c8cf941aa2" (UID: "b75ddf1c-5514-46f3-83b7-d0c8cf941aa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:36.417510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.417489    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-kube-api-access-v5khq" pod "b75ddf1c-5514-46f3-83b7-d0c8cf941aa2" (UID: "b75ddf1c-5514-46f3-83b7-d0c8cf941aa2"). InnerVolumeSpecName "kube-api-access-v5khq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:36.517926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.517889    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5khq\" (UniqueName: \"kubernetes.io/projected/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-kube-api-access-v5khq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:36.517926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:36.517921    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:37.126331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.126288    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"
Oct 30 19:23:37.126777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.126279    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g" event={"ID":"b75ddf1c-5514-46f3-83b7-d0c8cf941aa2","Type":"ContainerDied","Data":"605535ef10e1caaef57cb26b8d80e6b58ca912068839eaa294fc0ac68c71d5fb"}
Oct 30 19:23:37.126777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.126426    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="605535ef10e1caaef57cb26b8d80e6b58ca912068839eaa294fc0ac68c71d5fb"
Oct 30 19:23:37.238258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.238227    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:37.328657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.328631    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:37.430717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.430653    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-xks-ca\") pod \"f4d8959d-aca3-4c20-8558-3335f9b44729\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") "
Oct 30 19:23:37.430956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.430937    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d8959d-aca3-4c20-8558-3335f9b44729-kube-api-access-62fcl\" (UniqueName: \"kubernetes.io/projected/f4d8959d-aca3-4c20-8558-3335f9b44729-kube-api-access-62fcl\") pod \"f4d8959d-aca3-4c20-8558-3335f9b44729\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") "
Oct 30 19:23:37.431041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.430980    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-openshift-ca\") pod \"f4d8959d-aca3-4c20-8558-3335f9b44729\" (UID: \"f4d8959d-aca3-4c20-8558-3335f9b44729\") "
Oct 30 19:23:37.431394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.431368    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-openshift-ca" pod "f4d8959d-aca3-4c20-8558-3335f9b44729" (UID: "f4d8959d-aca3-4c20-8558-3335f9b44729"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:37.433480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.433453    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-xks-ca" pod "f4d8959d-aca3-4c20-8558-3335f9b44729" (UID: "f4d8959d-aca3-4c20-8558-3335f9b44729"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:37.433828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.433801    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d8959d-aca3-4c20-8558-3335f9b44729-kube-api-access-62fcl" pod "f4d8959d-aca3-4c20-8558-3335f9b44729" (UID: "f4d8959d-aca3-4c20-8558-3335f9b44729"). InnerVolumeSpecName "kube-api-access-62fcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:37.532670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.532640    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62fcl\" (UniqueName: \"kubernetes.io/projected/f4d8959d-aca3-4c20-8558-3335f9b44729-kube-api-access-62fcl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:37.532670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.532666    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:37.532833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.532676    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8959d-aca3-4c20-8558-3335f9b44729-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:37.846061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:37.845912    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:38.135429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:38.135333    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9" event={"ID":"f4d8959d-aca3-4c20-8558-3335f9b44729","Type":"ContainerDied","Data":"18d3266821faa0523d3a884f087e8c7cfa9639b385393a6d599ac1644f516559"}
Oct 30 19:23:38.135937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:38.135448    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18d3266821faa0523d3a884f087e8c7cfa9639b385393a6d599ac1644f516559"
Oct 30 19:23:38.135937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:38.135537    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"
Oct 30 19:23:38.651842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:38.651806    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:39.359782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:39.359743    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:39.988246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:39.988212    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:40.767493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:40.767462    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:41.351651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:41.351619    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:41.940198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:41.940161    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:42.719754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:42.719669    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:43.360667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:43.360632    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:44.120055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:44.120023    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:45.030711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:45.030337    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:46.168083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.166867    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.190581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.189818    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.190581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.189943    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:23:46.190581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.190261    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.291809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.291761    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:23:46.307053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.307006    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.330712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.321770    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.596767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.593776    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.596767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.595015    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.657711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.656785    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.732711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.731784    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:46.958714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:46.951346    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.024404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.023400    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.148906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.148841    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.169812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.169767    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.190847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.190797    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.195070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.191095    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.196594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.195764    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:23:47.196850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.196835    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:23:47.314782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.292797    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.319716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.317778    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.340139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.340091    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.477091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.472546    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:23:47.477091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.472610    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:23:47.477091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.472667    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:23:47.642782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.614823    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:23:47.642782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.616370    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:23:47.642782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.639885    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.657717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.643236    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:23:47.733219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.712607    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:23:47.733219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.712952    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:23:47.738735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.733709    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.768911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.739126    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:23:47.953929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.953838    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:47.955750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:47.955725    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:23:48.026831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.026785    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.066044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.065867    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.073760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.066820    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:23:48.176714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.159922    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.176714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.165805    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.176714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.166079    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:23:48.252825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.224846    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.252825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.242777    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.293807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.293763    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.320854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.320816    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.351118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.351079    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:23:48.632942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.631963    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.647444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.641230    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.713981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.712891    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.752729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.734069    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:23:48.816714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.806873    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:23:48.816714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.806936    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:48.816714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.815345    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:48.909861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:48.909818    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": context deadline exceeded"
Oct 30 19:23:49.105342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.105297    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.124836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.119750    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:23:49.147780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.142273    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:49.162945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.162836    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.163937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.163915    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:23:49.177606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.176501    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.187751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.187483    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.252996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.252933    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.355749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.352266    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:23:49.370261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.359116    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.370261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.369133    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:23:49.446828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.446729    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.645012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.644951    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.645589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.645558    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:23:49.719283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.719163    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:49.756713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:49.744775    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.089776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.083091    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:23:50.127356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.125893    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.134726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.127888    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.171064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.170994    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.197734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.190771    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.254858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.254797    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.360847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.360745    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.374338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.361079    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.400171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.400113    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.461739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.460802    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:23:50.656021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.646799    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:23:50.656021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.647869    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:23:50.745984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.721804    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:23:50.807412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.807380    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"]
Oct 30 19:23:50.808501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.808475    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f4d8959d-aca3-4c20-8558-3335f9b44729" containerName="revoke-keys"
Oct 30 19:23:50.808668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.808652    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d8959d-aca3-4c20-8558-3335f9b44729" containerName="revoke-keys"
Oct 30 19:23:50.808836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.808821    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b75ddf1c-5514-46f3-83b7-d0c8cf941aa2" containerName="revoke-keys"
Oct 30 19:23:50.808929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.808917    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75ddf1c-5514-46f3-83b7-d0c8cf941aa2" containerName="revoke-keys"
Oct 30 19:23:50.809189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.809174    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d8959d-aca3-4c20-8558-3335f9b44729" containerName="revoke-keys"
Oct 30 19:23:50.809303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.809292    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75ddf1c-5514-46f3-83b7-d0c8cf941aa2" containerName="revoke-keys"
Oct 30 19:23:50.856869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.853928    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:50.869997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.869970    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:23:50.908975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.908908    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:23:50.931725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.931682    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:23:50.957737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.957290    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:50.957737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.957364    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306e5049-a83d-4e23-bde7-66c10dfe4a63-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:50.966953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.966925    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:23:50.982780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.982750    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:50.982957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.982832    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:50.982957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.982903    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:50.982957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.982947    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:50.983144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.983083    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:50.983201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.983145    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:50.983256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:50.983238    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cs9x\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-kube-api-access-6cs9x\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.029227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.029200    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:23:51.029469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.029459    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:23:51.034825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.034805    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"]
Oct 30 19:23:51.036283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.036258    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:23:51.038884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.038864    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:23:51.040339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.040315    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:23:51.040958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.040937    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:23:51.053830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.053808    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:23:51.057796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.057762    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:23:51.085001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.084969    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.085164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.085027    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.085164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.085088    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.085297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.085241    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.085297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.085283    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.085406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.085369    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cs9x\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-kube-api-access-6cs9x\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.085467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.085435    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-data\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.085530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.085473    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.085593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.085532    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306e5049-a83d-4e23-bde7-66c10dfe4a63-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.085649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.085600    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.086311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.086278    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-credential-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.087183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.086706    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-socket\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.087183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.087146    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-certs\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.087538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.087516    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306e5049-a83d-4e23-bde7-66c10dfe4a63-istiod-ca-cert\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.089827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.089801    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-envoy\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.090330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.090275    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-podinfo\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.102711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.102461    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-token\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.106730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.103335    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cs9x\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-kube-api-access-6cs9x\") pod \"e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") " pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.265919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.265781    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:51.562028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.561995    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"]
Oct 30 19:23:51.562937 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:51.562904    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306e5049_a83d_4e23_bde7_66c10dfe4a63.slice/crio-392fced859e8f3fe87c01d16430a524458e8899690f5498546d23aaf223ddd64 WatchSource:0}: Error finding container 392fced859e8f3fe87c01d16430a524458e8899690f5498546d23aaf223ddd64: Status 404 returned error can't find the container with id 392fced859e8f3fe87c01d16430a524458e8899690f5498546d23aaf223ddd64
Oct 30 19:23:51.565914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.565882    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:51.565998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.565947    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:51.565998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.565978    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:23:51.579057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.579030    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:51.791697 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.791663    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" event={"ID":"306e5049-a83d-4e23-bde7-66c10dfe4a63","Type":"ContainerStarted","Data":"720b5bda5b6a78b8f611061dfc2fa514195ee10ec322e6071d93cc4f0e94c6c9"}
Oct 30 19:23:51.791911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.791748    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" event={"ID":"306e5049-a83d-4e23-bde7-66c10dfe4a63","Type":"ContainerStarted","Data":"392fced859e8f3fe87c01d16430a524458e8899690f5498546d23aaf223ddd64"}
Oct 30 19:23:51.818702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:51.818582    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podStartSLOduration=1.818567679 podStartE2EDuration="1.818567679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:51.813830439 +0000 UTC m=+1066.716843487" watchObservedRunningTime="2026-10-30 19:23:51.818567679 +0000 UTC m=+1066.721580716"
Oct 30 19:23:52.140662 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:23:52.140632    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:23:52.266515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.266485    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:52.267767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.267743    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:23:52.267911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.267787    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:23:52.429091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.429000    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"]
Oct 30 19:23:52.433455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.433433    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.440161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.439393    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:52.443483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.443401    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:23:52.449677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.449167    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"]
Oct 30 19:23:52.529715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.517225    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:52.531043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.531007    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gr2q\" (UniqueName: \"kubernetes.io/projected/5de2891d-1fc5-4202-892a-a79fec28a009-kube-api-access-9gr2q\") pod \"maas-api-revoke-sub-5848a1f7-q4h9g\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.531319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.531297    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5848a1f7-q4h9g\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.531418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.531393    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5848a1f7-q4h9g\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.633477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.632811    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5848a1f7-q4h9g\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.633477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.632917    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5848a1f7-q4h9g\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.633477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.633076    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gr2q\" (UniqueName: \"kubernetes.io/projected/5de2891d-1fc5-4202-892a-a79fec28a009-kube-api-access-9gr2q\") pod \"maas-api-revoke-sub-5848a1f7-q4h9g\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.634339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.634312    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5848a1f7-q4h9g\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.638112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.638092    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5848a1f7-q4h9g\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.645605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.645582    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gr2q\" (UniqueName: \"kubernetes.io/projected/5de2891d-1fc5-4202-892a-a79fec28a009-kube-api-access-9gr2q\") pod \"maas-api-revoke-sub-5848a1f7-q4h9g\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:52.759936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:52.759858    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:53.070029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:53.070000    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"]
Oct 30 19:23:53.070344 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:53.070318    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de2891d_1fc5_4202_892a_a79fec28a009.slice/crio-1ae99508a423ec0d27f9783b55b80882fa3e14b7f4e0f300eb65fb4cc7ea59c4 WatchSource:0}: Error finding container 1ae99508a423ec0d27f9783b55b80882fa3e14b7f4e0f300eb65fb4cc7ea59c4: Status 404 returned error can't find the container with id 1ae99508a423ec0d27f9783b55b80882fa3e14b7f4e0f300eb65fb4cc7ea59c4
Oct 30 19:23:53.276821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:53.276774    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:23:53.277261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:53.276841    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:23:53.277261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:53.277011    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:53.837372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:53.837338    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g" event={"ID":"5de2891d-1fc5-4202-892a-a79fec28a009","Type":"ContainerStarted","Data":"496757f23390fcca8b0b9a00706bab1eb9f867364cfc42c3c73e91dfb5e35adf"}
Oct 30 19:23:53.837564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:53.837397    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g" event={"ID":"5de2891d-1fc5-4202-892a-a79fec28a009","Type":"ContainerStarted","Data":"1ae99508a423ec0d27f9783b55b80882fa3e14b7f4e0f300eb65fb4cc7ea59c4"}
Oct 30 19:23:53.855948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:53.855908    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g" podStartSLOduration=1.855893525 podStartE2EDuration="1.855893525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:53.854257431 +0000 UTC m=+1068.757270469" watchObservedRunningTime="2026-10-30 19:23:53.855893525 +0000 UTC m=+1068.758906561"
Oct 30 19:23:54.060796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:54.058898    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:54.266092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:54.266062    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:23:54.266261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:54.266112    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:23:54.846814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:54.846777    4925 generic.go:356] "Generic (PLEG): container finished" podID="5de2891d-1fc5-4202-892a-a79fec28a009" containerID="496757f23390fcca8b0b9a00706bab1eb9f867364cfc42c3c73e91dfb5e35adf" exitCode=0
Oct 30 19:23:54.847258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:54.846872    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g" event={"ID":"5de2891d-1fc5-4202-892a-a79fec28a009","Type":"ContainerDied","Data":"496757f23390fcca8b0b9a00706bab1eb9f867364cfc42c3c73e91dfb5e35adf"}
Oct 30 19:23:54.987000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:54.986655    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:55.266513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.266481    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:23:55.266716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.266538    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:23:55.640438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.640400    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"]
Oct 30 19:23:55.661773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.658701    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:55.689182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.689012    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:55.733030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.732990    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"]
Oct 30 19:23:55.788505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.788472    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ec99-55f4-42df-9e38-702359356667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd981257-wwdsp\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:55.788678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.788518    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvgq\" (UniqueName: \"kubernetes.io/projected/9c02ec99-55f4-42df-9e38-702359356667-kube-api-access-dnvgq\") pod \"maas-api-revoke-sub-cd981257-wwdsp\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:55.788771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.788748    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02ec99-55f4-42df-9e38-702359356667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd981257-wwdsp\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:55.890023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.889989    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ec99-55f4-42df-9e38-702359356667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd981257-wwdsp\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:55.890481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.890039    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvgq\" (UniqueName: \"kubernetes.io/projected/9c02ec99-55f4-42df-9e38-702359356667-kube-api-access-dnvgq\") pod \"maas-api-revoke-sub-cd981257-wwdsp\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:55.890481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.890158    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02ec99-55f4-42df-9e38-702359356667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd981257-wwdsp\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:55.890648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.890627    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ec99-55f4-42df-9e38-702359356667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd981257-wwdsp\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:55.892702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.892666    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02ec99-55f4-42df-9e38-702359356667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd981257-wwdsp\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:55.929181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:55.929142    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvgq\" (UniqueName: \"kubernetes.io/projected/9c02ec99-55f4-42df-9e38-702359356667-kube-api-access-dnvgq\") pod \"maas-api-revoke-sub-cd981257-wwdsp\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:56.047794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.047764    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:56.118714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.100753    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:56.133055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.133024    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:56.199887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.194801    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de2891d-1fc5-4202-892a-a79fec28a009-kube-api-access-9gr2q\" (UniqueName: \"kubernetes.io/projected/5de2891d-1fc5-4202-892a-a79fec28a009-kube-api-access-9gr2q\") pod \"5de2891d-1fc5-4202-892a-a79fec28a009\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") "
Oct 30 19:23:56.199887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.194853    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-xks-ca\") pod \"5de2891d-1fc5-4202-892a-a79fec28a009\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") "
Oct 30 19:23:56.199887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.194968    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-openshift-ca\") pod \"5de2891d-1fc5-4202-892a-a79fec28a009\" (UID: \"5de2891d-1fc5-4202-892a-a79fec28a009\") "
Oct 30 19:23:56.220714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.220177    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-openshift-ca" pod "5de2891d-1fc5-4202-892a-a79fec28a009" (UID: "5de2891d-1fc5-4202-892a-a79fec28a009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:56.223277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.223233    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-xks-ca" pod "5de2891d-1fc5-4202-892a-a79fec28a009" (UID: "5de2891d-1fc5-4202-892a-a79fec28a009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:56.225105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.224845    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de2891d-1fc5-4202-892a-a79fec28a009-kube-api-access-9gr2q" pod "5de2891d-1fc5-4202-892a-a79fec28a009" (UID: "5de2891d-1fc5-4202-892a-a79fec28a009"). InnerVolumeSpecName "kube-api-access-9gr2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:56.266063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.266039    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:23:56.266412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.266084    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:23:56.318713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.310888    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gr2q\" (UniqueName: \"kubernetes.io/projected/5de2891d-1fc5-4202-892a-a79fec28a009-kube-api-access-9gr2q\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:56.318713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.310927    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:56.318713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.310942    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de2891d-1fc5-4202-892a-a79fec28a009-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:56.356913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.356886    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"]
Oct 30 19:23:56.357901 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:23:56.357873    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c02ec99_55f4_42df_9e38_702359356667.slice/crio-99f6db88d851d94b42777b13f42882dc8838d6dd286a5d94d814343f39d9af02 WatchSource:0}: Error finding container 99f6db88d851d94b42777b13f42882dc8838d6dd286a5d94d814343f39d9af02: Status 404 returned error can't find the container with id 99f6db88d851d94b42777b13f42882dc8838d6dd286a5d94d814343f39d9af02
Oct 30 19:23:56.493639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.493534    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:56.862541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.862454    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g" event={"ID":"5de2891d-1fc5-4202-892a-a79fec28a009","Type":"ContainerDied","Data":"1ae99508a423ec0d27f9783b55b80882fa3e14b7f4e0f300eb65fb4cc7ea59c4"}
Oct 30 19:23:56.862541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.862474    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"
Oct 30 19:23:56.862541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.862504    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae99508a423ec0d27f9783b55b80882fa3e14b7f4e0f300eb65fb4cc7ea59c4"
Oct 30 19:23:56.864145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.864120    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp" event={"ID":"9c02ec99-55f4-42df-9e38-702359356667","Type":"ContainerStarted","Data":"7220a3a96a113ebf9267af728a1c195d05e20cefd3026b814fd1c084b5523800"}
Oct 30 19:23:56.864222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.864156    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp" event={"ID":"9c02ec99-55f4-42df-9e38-702359356667","Type":"ContainerStarted","Data":"99f6db88d851d94b42777b13f42882dc8838d6dd286a5d94d814343f39d9af02"}
Oct 30 19:23:56.885126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.885089    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp" podStartSLOduration=1.885067091 podStartE2EDuration="1.885067091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:56.880600927 +0000 UTC m=+1071.783613965" watchObservedRunningTime="2026-10-30 19:23:56.885067091 +0000 UTC m=+1071.788080127"
Oct 30 19:23:56.893951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:56.893917    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:57.221757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:57.217900    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:57.266893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:57.266856    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:23:57.267063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:57.266906    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:23:57.872868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:57.872833    4925 generic.go:356] "Generic (PLEG): container finished" podID="9c02ec99-55f4-42df-9e38-702359356667" containerID="7220a3a96a113ebf9267af728a1c195d05e20cefd3026b814fd1c084b5523800" exitCode=0
Oct 30 19:23:57.873045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:57.872914    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp" event={"ID":"9c02ec99-55f4-42df-9e38-702359356667","Type":"ContainerDied","Data":"7220a3a96a113ebf9267af728a1c195d05e20cefd3026b814fd1c084b5523800"}
Oct 30 19:23:58.213292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:58.213252    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:58.273597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:58.266469    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:23:58.273597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:58.266525    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 19:23:59.079304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.079278    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:59.165662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.165628    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c02ec99-55f4-42df-9e38-702359356667-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02ec99-55f4-42df-9e38-702359356667-maas-api-xks-ca\") pod \"9c02ec99-55f4-42df-9e38-702359356667\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") "
Oct 30 19:23:59.165863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.165676    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c02ec99-55f4-42df-9e38-702359356667-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ec99-55f4-42df-9e38-702359356667-maas-api-openshift-ca\") pod \"9c02ec99-55f4-42df-9e38-702359356667\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") "
Oct 30 19:23:59.165912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.165886    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c02ec99-55f4-42df-9e38-702359356667-kube-api-access-dnvgq\" (UniqueName: \"kubernetes.io/projected/9c02ec99-55f4-42df-9e38-702359356667-kube-api-access-dnvgq\") pod \"9c02ec99-55f4-42df-9e38-702359356667\" (UID: \"9c02ec99-55f4-42df-9e38-702359356667\") "
Oct 30 19:23:59.166138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.166113    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c02ec99-55f4-42df-9e38-702359356667-maas-api-openshift-ca" pod "9c02ec99-55f4-42df-9e38-702359356667" (UID: "9c02ec99-55f4-42df-9e38-702359356667"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:59.166571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.166523    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ec99-55f4-42df-9e38-702359356667-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:59.168015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.167985    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c02ec99-55f4-42df-9e38-702359356667-maas-api-xks-ca" pod "9c02ec99-55f4-42df-9e38-702359356667" (UID: "9c02ec99-55f4-42df-9e38-702359356667"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:59.168193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.168167    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c02ec99-55f4-42df-9e38-702359356667-kube-api-access-dnvgq" pod "9c02ec99-55f4-42df-9e38-702359356667" (UID: "9c02ec99-55f4-42df-9e38-702359356667"). InnerVolumeSpecName "kube-api-access-dnvgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:59.206761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.206674    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:59.276455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.276418    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvgq\" (UniqueName: \"kubernetes.io/projected/9c02ec99-55f4-42df-9e38-702359356667-kube-api-access-dnvgq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:59.276455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.276457    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c02ec99-55f4-42df-9e38-702359356667-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:59.282097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.282068    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:59.283397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.283371    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:59.284185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.284162    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:23:59.893732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.892710    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"
Oct 30 19:23:59.893732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.892729    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp" event={"ID":"9c02ec99-55f4-42df-9e38-702359356667","Type":"ContainerDied","Data":"99f6db88d851d94b42777b13f42882dc8838d6dd286a5d94d814343f39d9af02"}
Oct 30 19:23:59.893732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.892783    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f6db88d851d94b42777b13f42882dc8838d6dd286a5d94d814343f39d9af02"
Oct 30 19:23:59.994259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:23:59.994223    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:00.751183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:00.751151    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:01.435766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:01.435735    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:02.795716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:02.791745    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:03.054007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.053903    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.091088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.091043    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.111310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.111264    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.183715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.178809    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.280902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.280847    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.328923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.328819    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.356575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.356532    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.377827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.377778    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:03.462829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.462777    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.652843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.652795    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.654275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.653108    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:03.655356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.653139    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:24:03.749843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:03.749792    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.053924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.053802    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"]
Oct 30 19:24:04.056635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.054533    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.057909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.057888    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9c02ec99-55f4-42df-9e38-702359356667" containerName="revoke-keys"
Oct 30 19:24:04.058072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.058057    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c02ec99-55f4-42df-9e38-702359356667" containerName="revoke-keys"
Oct 30 19:24:04.058229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.058216    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5de2891d-1fc5-4202-892a-a79fec28a009" containerName="revoke-keys"
Oct 30 19:24:04.058353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.058339    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de2891d-1fc5-4202-892a-a79fec28a009" containerName="revoke-keys"
Oct 30 19:24:04.058672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.058647    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c02ec99-55f4-42df-9e38-702359356667" containerName="revoke-keys"
Oct 30 19:24:04.058834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.058820    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de2891d-1fc5-4202-892a-a79fec28a009" containerName="revoke-keys"
Oct 30 19:24:04.091223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.091180    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:24:04.093805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.091827    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.094608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.092602    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.095924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.095909    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:24:04.106720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.102210    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:04.106720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.103228    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:24:04.106720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.105790    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"]
Oct 30 19:24:04.120250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.116038    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.120250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.116753    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:24:04.227723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.220406    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bc2\" (UniqueName: \"kubernetes.io/projected/02e39e4f-2be6-4c2e-a696-f01471a5140f-kube-api-access-s7bc2\") pod \"maas-api-revoke-sub-8d0c1c42-7rxh4\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.227723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.220468    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d0c1c42-7rxh4\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.227723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.220577    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d0c1c42-7rxh4\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.243845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.228111    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:24:04.243845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.233410    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:24:04.317792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.311352    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.317792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.311781    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:24:04.320991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.318842    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:24:04.382714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.374630    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bc2\" (UniqueName: \"kubernetes.io/projected/02e39e4f-2be6-4c2e-a696-f01471a5140f-kube-api-access-s7bc2\") pod \"maas-api-revoke-sub-8d0c1c42-7rxh4\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.382714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.374735    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d0c1c42-7rxh4\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.382714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.374838    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d0c1c42-7rxh4\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.407012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.405741    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.407012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.405918    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d0c1c42-7rxh4\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.407012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.406119    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:24:04.466724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.457126    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d0c1c42-7rxh4\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.467296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.467256    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:04.522917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.522885    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:24:04.524478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.524445    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.525022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.525000    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:24:04.560535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.552514    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bc2\" (UniqueName: \"kubernetes.io/projected/02e39e4f-2be6-4c2e-a696-f01471a5140f-kube-api-access-s7bc2\") pod \"maas-api-revoke-sub-8d0c1c42-7rxh4\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.585070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.584992    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:04.585525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.585491    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.816811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.816767    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:24:04.841924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.829870    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:24:04.859900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.817143    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:04.871657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.869713    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:24:04.924906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.924876    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:24:04.925210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.925191    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:24:04.925402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:04.925387    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:24:05.066717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.059099    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.102714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.102592    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.130892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.130581    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:24:05.131805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.131681    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:24:05.220110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.220061    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Oct 30 19:24:05.228917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.228878    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:24:05.424710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.420321    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.430052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.430017    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:05.439836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.439778    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:24:05.538974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.538925    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.597987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.543675    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.839770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.814578    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:24:05.893715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.846114    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.923928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.851705    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.924717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.869246    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.948123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.880935    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:05.952023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:05.946533    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.106776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.104768    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:24:06.156137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.150751    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.205893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.196983    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.205893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.200742    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:24:06.248631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.248484    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.570712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.555016    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.602806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.602759    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:24:06.693898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.693797    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:06.929714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.846955    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.929714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.878503    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:06.929714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.878576    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.929714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.882256    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.929714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.929705    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.931286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.931255    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:24:06.970272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.970159    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:06.991070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:06.970515    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:24:07.033990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.032375    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:24:07.131715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.107903    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:07.149364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.129868    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:24:07.218908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.218863    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:07.219263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.219236    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:24:07.220072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.220047    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:24:07.296286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.295799    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:07.502757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.502733    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:24:07.502973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.502958    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"]
Oct 30 19:24:07.510832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.510322    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:24:07.510832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.510351    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.511213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.511031    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:24:07.511213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.511098    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:24:07.511213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.511129    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:24:07.511213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.511147    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"]
Oct 30 19:24:07.511449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.511241    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:24:07.511449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.511306    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:24:07.511449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.511352    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:24:07.511449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.511388    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:24:07.511449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.511420    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:24:07.543667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.543599    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"]
Oct 30 19:24:07.591287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.591206    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57874b60-n9z8g\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.591463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.591382    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57874b60-n9z8g\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.591463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.591412    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7krm\" (UniqueName: \"kubernetes.io/projected/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-kube-api-access-b7krm\") pod \"maas-api-revoke-sub-57874b60-n9z8g\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.626121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.626047    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4" event={"ID":"02e39e4f-2be6-4c2e-a696-f01471a5140f","Type":"ContainerStarted","Data":"58ee90026aa6b22daf81bebe8dcdf8b5dc0a505991b977e4f993756e48b4d8b7"}
Oct 30 19:24:07.692471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.692438    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57874b60-n9z8g\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.692642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.692565    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57874b60-n9z8g\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.692642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.692587    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7krm\" (UniqueName: \"kubernetes.io/projected/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-kube-api-access-b7krm\") pod \"maas-api-revoke-sub-57874b60-n9z8g\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.693232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.693209    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57874b60-n9z8g\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.696221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.696199    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57874b60-n9z8g\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.707115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.707091    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7krm\" (UniqueName: \"kubernetes.io/projected/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-kube-api-access-b7krm\") pod \"maas-api-revoke-sub-57874b60-n9z8g\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:07.725194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.725166    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:07.851048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:07.850927    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:08.145508 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:24:08.145482    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b74efc3_2fa8_4460_98a5_4bf5c1c7223f.slice/crio-f1ace12297d6b2d63c4d39a47a8f68ed6ae55084ea97c815bcd6b1434684efcd WatchSource:0}: Error finding container f1ace12297d6b2d63c4d39a47a8f68ed6ae55084ea97c815bcd6b1434684efcd: Status 404 returned error can't find the container with id f1ace12297d6b2d63c4d39a47a8f68ed6ae55084ea97c815bcd6b1434684efcd
Oct 30 19:24:08.146073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:08.146045    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"]
Oct 30 19:24:08.542677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:08.542521    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:08.634916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:08.634882    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g" event={"ID":"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f","Type":"ContainerStarted","Data":"d4a03abdaef1973f9bf19141fab73482f78caa334a76ac27999fce5d5e95e6af"}
Oct 30 19:24:08.635113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:08.634938    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g" event={"ID":"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f","Type":"ContainerStarted","Data":"f1ace12297d6b2d63c4d39a47a8f68ed6ae55084ea97c815bcd6b1434684efcd"}
Oct 30 19:24:08.636608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:08.636554    4925 generic.go:356] "Generic (PLEG): container finished" podID="02e39e4f-2be6-4c2e-a696-f01471a5140f" containerID="579025e32da6698e02dcbc4a60a155a24cc45173b391dcbb42336a868b361e5e" exitCode=0
Oct 30 19:24:08.636788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:08.636639    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4" event={"ID":"02e39e4f-2be6-4c2e-a696-f01471a5140f","Type":"ContainerDied","Data":"579025e32da6698e02dcbc4a60a155a24cc45173b391dcbb42336a868b361e5e"}
Oct 30 19:24:08.654570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:08.654525    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g" podStartSLOduration=4.6545132559999995 podStartE2EDuration="4.654513256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:08.652191482 +0000 UTC m=+1083.555204519" watchObservedRunningTime="2026-10-30 19:24:08.654513256 +0000 UTC m=+1083.557526292"
Oct 30 19:24:09.351317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.351200    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:09.646458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.646430    4925 generic.go:356] "Generic (PLEG): container finished" podID="9b74efc3-2fa8-4460-98a5-4bf5c1c7223f" containerID="d4a03abdaef1973f9bf19141fab73482f78caa334a76ac27999fce5d5e95e6af" exitCode=0
Oct 30 19:24:09.646625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.646513    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g" event={"ID":"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f","Type":"ContainerDied","Data":"d4a03abdaef1973f9bf19141fab73482f78caa334a76ac27999fce5d5e95e6af"}
Oct 30 19:24:09.840385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.840363    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:09.929479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.929390    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-openshift-ca\") pod \"02e39e4f-2be6-4c2e-a696-f01471a5140f\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") "
Oct 30 19:24:09.929479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.929443    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e39e4f-2be6-4c2e-a696-f01471a5140f-kube-api-access-s7bc2\" (UniqueName: \"kubernetes.io/projected/02e39e4f-2be6-4c2e-a696-f01471a5140f-kube-api-access-s7bc2\") pod \"02e39e4f-2be6-4c2e-a696-f01471a5140f\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") "
Oct 30 19:24:09.929727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.929534    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-xks-ca\") pod \"02e39e4f-2be6-4c2e-a696-f01471a5140f\" (UID: \"02e39e4f-2be6-4c2e-a696-f01471a5140f\") "
Oct 30 19:24:09.929966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.929934    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-openshift-ca" pod "02e39e4f-2be6-4c2e-a696-f01471a5140f" (UID: "02e39e4f-2be6-4c2e-a696-f01471a5140f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:09.930255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.930238    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:09.932329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.932302    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-xks-ca" pod "02e39e4f-2be6-4c2e-a696-f01471a5140f" (UID: "02e39e4f-2be6-4c2e-a696-f01471a5140f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:09.932565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.932450    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e39e4f-2be6-4c2e-a696-f01471a5140f-kube-api-access-s7bc2" pod "02e39e4f-2be6-4c2e-a696-f01471a5140f" (UID: "02e39e4f-2be6-4c2e-a696-f01471a5140f"). InnerVolumeSpecName "kube-api-access-s7bc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:09.995104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:09.995071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:10.032208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.032180    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7bc2\" (UniqueName: \"kubernetes.io/projected/02e39e4f-2be6-4c2e-a696-f01471a5140f-kube-api-access-s7bc2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:10.032208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.032206    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e39e4f-2be6-4c2e-a696-f01471a5140f-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:10.655378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.655349    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"
Oct 30 19:24:10.655378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.655371    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4" event={"ID":"02e39e4f-2be6-4c2e-a696-f01471a5140f","Type":"ContainerDied","Data":"58ee90026aa6b22daf81bebe8dcdf8b5dc0a505991b977e4f993756e48b4d8b7"}
Oct 30 19:24:10.655872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.655445    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58ee90026aa6b22daf81bebe8dcdf8b5dc0a505991b977e4f993756e48b4d8b7"
Oct 30 19:24:10.746466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.746423    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:10.852884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.852858    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:10.944043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.943970    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-kube-api-access-b7krm\" (UniqueName: \"kubernetes.io/projected/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-kube-api-access-b7krm\") pod \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") "
Oct 30 19:24:10.944361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.944281    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-xks-ca\") pod \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") "
Oct 30 19:24:10.944506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.944382    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-openshift-ca\") pod \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\" (UID: \"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f\") "
Oct 30 19:24:10.944843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.944819    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-openshift-ca" pod "9b74efc3-2fa8-4460-98a5-4bf5c1c7223f" (UID: "9b74efc3-2fa8-4460-98a5-4bf5c1c7223f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:10.944989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.944970    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:10.946493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.946472    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-kube-api-access-b7krm" pod "9b74efc3-2fa8-4460-98a5-4bf5c1c7223f" (UID: "9b74efc3-2fa8-4460-98a5-4bf5c1c7223f"). InnerVolumeSpecName "kube-api-access-b7krm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:10.946574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:10.946521    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-xks-ca" pod "9b74efc3-2fa8-4460-98a5-4bf5c1c7223f" (UID: "9b74efc3-2fa8-4460-98a5-4bf5c1c7223f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:11.046855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:11.046826    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:11.046855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:11.046852    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7krm\" (UniqueName: \"kubernetes.io/projected/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f-kube-api-access-b7krm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:11.577198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:11.577110    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:11.664826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:11.664799    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"
Oct 30 19:24:11.664826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:11.664815    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g" event={"ID":"9b74efc3-2fa8-4460-98a5-4bf5c1c7223f","Type":"ContainerDied","Data":"f1ace12297d6b2d63c4d39a47a8f68ed6ae55084ea97c815bcd6b1434684efcd"}
Oct 30 19:24:11.665216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:11.664864    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1ace12297d6b2d63c4d39a47a8f68ed6ae55084ea97c815bcd6b1434684efcd"
Oct 30 19:24:12.216736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:12.216133    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:12.853022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:12.852771    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:13.612217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:13.612186    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:14.261581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:14.261547    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:15.141514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.141484    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:15.826519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.826481    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"]
Oct 30 19:24:15.827179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.827163    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9b74efc3-2fa8-4460-98a5-4bf5c1c7223f" containerName="revoke-keys"
Oct 30 19:24:15.827219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.827182    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b74efc3-2fa8-4460-98a5-4bf5c1c7223f" containerName="revoke-keys"
Oct 30 19:24:15.827219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.827212    4925 cpu_manager.go:414] "containerMap: removing container" podUID="02e39e4f-2be6-4c2e-a696-f01471a5140f" containerName="revoke-keys"
Oct 30 19:24:15.827219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.827218    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e39e4f-2be6-4c2e-a696-f01471a5140f" containerName="revoke-keys"
Oct 30 19:24:15.827334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.827309    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b74efc3-2fa8-4460-98a5-4bf5c1c7223f" containerName="revoke-keys"
Oct 30 19:24:15.827334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.827323    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e39e4f-2be6-4c2e-a696-f01471a5140f" containerName="revoke-keys"
Oct 30 19:24:15.831049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.831033    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:15.834239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.834020    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:15.834239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.834217    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:24:15.839646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.839353    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"]
Oct 30 19:24:15.883697 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.883662    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:15.917962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.917921    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg75c\" (UniqueName: \"kubernetes.io/projected/4359741e-e0c0-4204-b437-2d5a4147f65d-kube-api-access-jg75c\") pod \"maas-api-revoke-sub-f2df9cf0-swvlr\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:15.918235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.918214    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2df9cf0-swvlr\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:15.918338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:15.918285    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2df9cf0-swvlr\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:16.019593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:16.019560    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2df9cf0-swvlr\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:16.019782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:16.019616    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2df9cf0-swvlr\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:16.019782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:16.019748    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg75c\" (UniqueName: \"kubernetes.io/projected/4359741e-e0c0-4204-b437-2d5a4147f65d-kube-api-access-jg75c\") pod \"maas-api-revoke-sub-f2df9cf0-swvlr\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:16.020328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:16.020302    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2df9cf0-swvlr\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:16.022132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:16.022113    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2df9cf0-swvlr\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:16.031122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:16.031097    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg75c\" (UniqueName: \"kubernetes.io/projected/4359741e-e0c0-4204-b437-2d5a4147f65d-kube-api-access-jg75c\") pod \"maas-api-revoke-sub-f2df9cf0-swvlr\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:16.182919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:16.182865    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:16.482680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:16.482653    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"]
Oct 30 19:24:16.483747 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:24:16.483720    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4359741e_e0c0_4204_b437_2d5a4147f65d.slice/crio-ecb2cb14bffc0270d3811efc9fdad56045d44bf6ad132a8191330900f4b85270 WatchSource:0}: Error finding container ecb2cb14bffc0270d3811efc9fdad56045d44bf6ad132a8191330900f4b85270: Status 404 returned error can't find the container with id ecb2cb14bffc0270d3811efc9fdad56045d44bf6ad132a8191330900f4b85270
Oct 30 19:24:16.707682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:16.707649    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr" event={"ID":"4359741e-e0c0-4204-b437-2d5a4147f65d","Type":"ContainerStarted","Data":"ecb2cb14bffc0270d3811efc9fdad56045d44bf6ad132a8191330900f4b85270"}
Oct 30 19:24:17.087734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.087612    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:17.746358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.746263    4925 generic.go:356] "Generic (PLEG): container finished" podID="4359741e-e0c0-4204-b437-2d5a4147f65d" containerID="163e52568b64445dd1466cd5ec7a1b5731f4bcfbd6e7175e282e042a20a397d6" exitCode=0
Oct 30 19:24:17.746358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.746327    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr" event={"ID":"4359741e-e0c0-4204-b437-2d5a4147f65d","Type":"ContainerDied","Data":"163e52568b64445dd1466cd5ec7a1b5731f4bcfbd6e7175e282e042a20a397d6"}
Oct 30 19:24:17.749166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.749139    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:17.752321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.752293    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"]
Oct 30 19:24:17.758647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.757501    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:17.777850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.777682    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"]
Oct 30 19:24:17.844836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.844808    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06082bd8-mdpnf\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:17.845034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.845016    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06082bd8-mdpnf\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:17.845087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.845053    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjs6\" (UniqueName: \"kubernetes.io/projected/0378d84f-51a8-46fe-bd81-6976fa1ef734-kube-api-access-8hjs6\") pod \"maas-api-revoke-sub-06082bd8-mdpnf\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:17.946855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.946810    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06082bd8-mdpnf\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:17.947031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.946882    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjs6\" (UniqueName: \"kubernetes.io/projected/0378d84f-51a8-46fe-bd81-6976fa1ef734-kube-api-access-8hjs6\") pod \"maas-api-revoke-sub-06082bd8-mdpnf\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:17.947100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.947053    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06082bd8-mdpnf\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:17.947498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.947475    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06082bd8-mdpnf\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:17.950187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.950003    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06082bd8-mdpnf\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:17.965301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:17.965255    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjs6\" (UniqueName: \"kubernetes.io/projected/0378d84f-51a8-46fe-bd81-6976fa1ef734-kube-api-access-8hjs6\") pod \"maas-api-revoke-sub-06082bd8-mdpnf\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:18.094162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:18.094089    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:18.380124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:18.380091    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"]
Oct 30 19:24:18.381144 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:24:18.381117    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0378d84f_51a8_46fe_bd81_6976fa1ef734.slice/crio-09d84ce7d646b08c787960f502cb79b8bdb3a4134d6bf47fdd9c80a397afe503 WatchSource:0}: Error finding container 09d84ce7d646b08c787960f502cb79b8bdb3a4134d6bf47fdd9c80a397afe503: Status 404 returned error can't find the container with id 09d84ce7d646b08c787960f502cb79b8bdb3a4134d6bf47fdd9c80a397afe503
Oct 30 19:24:18.576308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:18.576271    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:18.752750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:18.752715    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:18.755375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:18.755345    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf" event={"ID":"0378d84f-51a8-46fe-bd81-6976fa1ef734","Type":"ContainerStarted","Data":"507e5a21945e641e3fd73d81234c016a5bbaa707412b5c24689b38c1cfe34ad5"}
Oct 30 19:24:18.755496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:18.755392    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf" event={"ID":"0378d84f-51a8-46fe-bd81-6976fa1ef734","Type":"ContainerStarted","Data":"09d84ce7d646b08c787960f502cb79b8bdb3a4134d6bf47fdd9c80a397afe503"}
Oct 30 19:24:18.785900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:18.785276    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf" podStartSLOduration=1.7852571350000002 podStartE2EDuration="1.785257135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:18.779997044 +0000 UTC m=+1093.683010082" watchObservedRunningTime="2026-10-30 19:24:18.785257135 +0000 UTC m=+1093.688270169"
Oct 30 19:24:18.931076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:18.931052    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:19.016922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.016838    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"]
Oct 30 19:24:19.017706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.017672    4925 cpu_manager.go:414] "containerMap: removing container" podUID="4359741e-e0c0-4204-b437-2d5a4147f65d" containerName="revoke-keys"
Oct 30 19:24:19.017832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.017736    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4359741e-e0c0-4204-b437-2d5a4147f65d" containerName="revoke-keys"
Oct 30 19:24:19.017939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.017925    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4359741e-e0c0-4204-b437-2d5a4147f65d" containerName="revoke-keys"
Oct 30 19:24:19.021912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.021894    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.027934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.027465    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-250fb057-openshift-default-dockercfg-rnvw6\""
Oct 30 19:24:19.030511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.030202    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"]
Oct 30 19:24:19.063486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.063461    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4359741e-e0c0-4204-b437-2d5a4147f65d-kube-api-access-jg75c\" (UniqueName: \"kubernetes.io/projected/4359741e-e0c0-4204-b437-2d5a4147f65d-kube-api-access-jg75c\") pod \"4359741e-e0c0-4204-b437-2d5a4147f65d\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") "
Oct 30 19:24:19.063614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.063508    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-xks-ca\") pod \"4359741e-e0c0-4204-b437-2d5a4147f65d\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") "
Oct 30 19:24:19.063725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.063647    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-openshift-ca\") pod \"4359741e-e0c0-4204-b437-2d5a4147f65d\" (UID: \"4359741e-e0c0-4204-b437-2d5a4147f65d\") "
Oct 30 19:24:19.064076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.064047    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-openshift-ca" pod "4359741e-e0c0-4204-b437-2d5a4147f65d" (UID: "4359741e-e0c0-4204-b437-2d5a4147f65d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:19.064287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.064267    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:19.066332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.066299    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4359741e-e0c0-4204-b437-2d5a4147f65d-kube-api-access-jg75c" pod "4359741e-e0c0-4204-b437-2d5a4147f65d" (UID: "4359741e-e0c0-4204-b437-2d5a4147f65d"). InnerVolumeSpecName "kube-api-access-jg75c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:19.066444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.066330    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-xks-ca" pod "4359741e-e0c0-4204-b437-2d5a4147f65d" (UID: "4359741e-e0c0-4204-b437-2d5a4147f65d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:19.113348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.112457    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:19.186712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.185755    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486ee0c9-a8fd-41b5-9d86-e521411aa661-istiod-ca-cert\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.186712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.185924    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-envoy\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.186712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.186154    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-podinfo\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.186712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.186421    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-data\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.186712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.186567    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-certs\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.186712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.186625    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thccp\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-kube-api-access-thccp\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.186712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.186658    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-socket\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.187278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.186802    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-credential-socket\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.187278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.186888    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-token\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.187278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.187086    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg75c\" (UniqueName: \"kubernetes.io/projected/4359741e-e0c0-4204-b437-2d5a4147f65d-kube-api-access-jg75c\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:19.187278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.187106    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4359741e-e0c0-4204-b437-2d5a4147f65d-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:19.289237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.289126    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486ee0c9-a8fd-41b5-9d86-e521411aa661-istiod-ca-cert\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.289456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.289437    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-envoy\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.289679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.289663    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-podinfo\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.289882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.289865    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-data\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.290055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.290038    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-certs\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.290178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.290165    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thccp\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-kube-api-access-thccp\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.290282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.290267    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-socket\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.290419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.290404    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-credential-socket\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.290543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.290530    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-token\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.291878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.291847    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-certs\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.292375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.292352    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486ee0c9-a8fd-41b5-9d86-e521411aa661-istiod-ca-cert\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.293033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.293011    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-socket\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.293394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.293376    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-credential-socket\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.295084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.295058    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-data\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.296912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.296167    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-envoy\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.300109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.299357    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:19.303610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.302544    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-podinfo\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.315150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.314664    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thccp\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-kube-api-access-thccp\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.315150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.315105    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-token\") pod \"e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") " pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.341642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.341612    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:19.641232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.641194    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:19.656980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.656949    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"]
Oct 30 19:24:19.659325 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:24:19.659296    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486ee0c9_a8fd_41b5_9d86_e521411aa661.slice/crio-e2c15f5963bb2f3321e7aa3d78867fffe17d6718f01589cb19d098b9bd55908d WatchSource:0}: Error finding container e2c15f5963bb2f3321e7aa3d78867fffe17d6718f01589cb19d098b9bd55908d: Status 404 returned error can't find the container with id e2c15f5963bb2f3321e7aa3d78867fffe17d6718f01589cb19d098b9bd55908d
Oct 30 19:24:19.663264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.662956    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:24:19.663264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.663041    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:24:19.663264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.663084    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:24:19.769001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.768969    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh" event={"ID":"486ee0c9-a8fd-41b5-9d86-e521411aa661","Type":"ContainerStarted","Data":"47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a"}
Oct 30 19:24:19.769339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.769314    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh" event={"ID":"486ee0c9-a8fd-41b5-9d86-e521411aa661","Type":"ContainerStarted","Data":"e2c15f5963bb2f3321e7aa3d78867fffe17d6718f01589cb19d098b9bd55908d"}
Oct 30 19:24:19.770880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.770852    4925 generic.go:356] "Generic (PLEG): container finished" podID="0378d84f-51a8-46fe-bd81-6976fa1ef734" containerID="507e5a21945e641e3fd73d81234c016a5bbaa707412b5c24689b38c1cfe34ad5" exitCode=0
Oct 30 19:24:19.770977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.770903    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf" event={"ID":"0378d84f-51a8-46fe-bd81-6976fa1ef734","Type":"ContainerDied","Data":"507e5a21945e641e3fd73d81234c016a5bbaa707412b5c24689b38c1cfe34ad5"}
Oct 30 19:24:19.772364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.772340    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr" event={"ID":"4359741e-e0c0-4204-b437-2d5a4147f65d","Type":"ContainerDied","Data":"ecb2cb14bffc0270d3811efc9fdad56045d44bf6ad132a8191330900f4b85270"}
Oct 30 19:24:19.772477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.772375    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecb2cb14bffc0270d3811efc9fdad56045d44bf6ad132a8191330900f4b85270"
Oct 30 19:24:19.772477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.772424    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"
Oct 30 19:24:19.798609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:19.798556    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh" podStartSLOduration=1.798542683 podStartE2EDuration="1.798542683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:19.792724291 +0000 UTC m=+1094.695737361" watchObservedRunningTime="2026-10-30 19:24:19.798542683 +0000 UTC m=+1094.701555720"
Oct 30 19:24:20.342165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:20.342136    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:20.348666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:20.348638    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:20.697654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:20.697617    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:20.782434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:20.782026    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:20.782718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:20.782671    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:24:20.905797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:20.905345    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:21.002852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.002824    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:21.022755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.021703    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0378d84f-51a8-46fe-bd81-6976fa1ef734-kube-api-access-8hjs6\" (UniqueName: \"kubernetes.io/projected/0378d84f-51a8-46fe-bd81-6976fa1ef734-kube-api-access-8hjs6\") pod \"0378d84f-51a8-46fe-bd81-6976fa1ef734\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") "
Oct 30 19:24:21.022755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.021843    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-openshift-ca\") pod \"0378d84f-51a8-46fe-bd81-6976fa1ef734\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") "
Oct 30 19:24:21.022755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.022034    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-xks-ca\") pod \"0378d84f-51a8-46fe-bd81-6976fa1ef734\" (UID: \"0378d84f-51a8-46fe-bd81-6976fa1ef734\") "
Oct 30 19:24:21.023286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.022763    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-openshift-ca" pod "0378d84f-51a8-46fe-bd81-6976fa1ef734" (UID: "0378d84f-51a8-46fe-bd81-6976fa1ef734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:21.026365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.026258    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0378d84f-51a8-46fe-bd81-6976fa1ef734-kube-api-access-8hjs6" pod "0378d84f-51a8-46fe-bd81-6976fa1ef734" (UID: "0378d84f-51a8-46fe-bd81-6976fa1ef734"). InnerVolumeSpecName "kube-api-access-8hjs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:21.029766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.029734    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-xks-ca" pod "0378d84f-51a8-46fe-bd81-6976fa1ef734" (UID: "0378d84f-51a8-46fe-bd81-6976fa1ef734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:21.087530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.087487    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:21.123990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.123964    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:21.123990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.123987    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hjs6\" (UniqueName: \"kubernetes.io/projected/0378d84f-51a8-46fe-bd81-6976fa1ef734-kube-api-access-8hjs6\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:21.124124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.123999    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0378d84f-51a8-46fe-bd81-6976fa1ef734-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:21.270564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.270540    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:21.479941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.479907    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:21.857726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.839475    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"
Oct 30 19:24:21.857726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.845899    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf" event={"ID":"0378d84f-51a8-46fe-bd81-6976fa1ef734","Type":"ContainerDied","Data":"09d84ce7d646b08c787960f502cb79b8bdb3a4134d6bf47fdd9c80a397afe503"}
Oct 30 19:24:21.857726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.845963    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09d84ce7d646b08c787960f502cb79b8bdb3a4134d6bf47fdd9c80a397afe503"
Oct 30 19:24:21.883636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.883389    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:21.907788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.901821    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"]
Oct 30 19:24:21.907788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.902793    4925 cpu_manager.go:414] "containerMap: removing container" podUID="0378d84f-51a8-46fe-bd81-6976fa1ef734" containerName="revoke-keys"
Oct 30 19:24:21.907788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.902812    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0378d84f-51a8-46fe-bd81-6976fa1ef734" containerName="revoke-keys"
Oct 30 19:24:21.907788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.903029    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0378d84f-51a8-46fe-bd81-6976fa1ef734" containerName="revoke-keys"
Oct 30 19:24:21.916314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.912812    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:21.919566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:21.919489    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"]
Oct 30 19:24:22.003730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.003666    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e308e736-gw-openshift-default-dockercfg-6mrl9\""
Oct 30 19:24:22.076753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.075961    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-envoy\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.076753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.076101    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-token\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.076753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.076135    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istiod-ca-cert\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.076753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.076177    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-certs\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.076753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.076218    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-podinfo\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.076753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.076264    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-credential-socket\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.076753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.076416    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-data\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.076753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.076482    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h46g\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-kube-api-access-8h46g\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.076753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.076536    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-socket\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.088298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.088267    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:22.141785 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:24:22.141250    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:24:22.178078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.177811    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-data\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.177900    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h46g\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-kube-api-access-8h46g\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.177955    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-socket\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.178003    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-envoy\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.178122    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-token\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.178149    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istiod-ca-cert\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.178189    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-certs\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.178223    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-podinfo\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.178240    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-data\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.178311    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-credential-socket\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.178660    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-credential-socket\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.178927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.178897    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-socket\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.179081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.179061    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-certs\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.179333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.179311    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istiod-ca-cert\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.181260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.181242    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-envoy\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.182190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.182169    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-podinfo\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.190503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.190483    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-token\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.190600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.190581    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h46g\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-kube-api-access-8h46g\") pod \"e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") " pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.234850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.234817    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:22.560493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.560315    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:22.575981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.575503    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"]
Oct 30 19:24:22.580859 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:24:22.580836    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf711c427_67a2_4e2c_8d39_017ae8b90ac6.slice/crio-924d82f9db79150293de83c7e175b5071fde98370b06ea7f4a8f74aa3130a171 WatchSource:0}: Error finding container 924d82f9db79150293de83c7e175b5071fde98370b06ea7f4a8f74aa3130a171: Status 404 returned error can't find the container with id 924d82f9db79150293de83c7e175b5071fde98370b06ea7f4a8f74aa3130a171
Oct 30 19:24:22.583614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.583580    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:24:22.583758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.583734    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:24:22.583837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.583794    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:24:22.730567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.730536    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:22.915274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.912461    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" event={"ID":"f711c427-67a2-4e2c-8d39-017ae8b90ac6","Type":"ContainerStarted","Data":"6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9"}
Oct 30 19:24:22.915274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.912517    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" event={"ID":"f711c427-67a2-4e2c-8d39-017ae8b90ac6","Type":"ContainerStarted","Data":"924d82f9db79150293de83c7e175b5071fde98370b06ea7f4a8f74aa3130a171"}
Oct 30 19:24:22.955888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:22.955829    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" podStartSLOduration=1.955811752 podStartE2EDuration="1.955811752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:22.934359712 +0000 UTC m=+1097.837372753" watchObservedRunningTime="2026-10-30 19:24:22.955811752 +0000 UTC m=+1097.858824790"
Oct 30 19:24:23.235487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:23.235400    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:23.237136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:23.237106    4925 patch_prober.go:29] interesting pod/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:24:23.237288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:23.237167    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:24:23.451951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:23.451904    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:24.239850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:24.239815    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:24.327852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:24.327814    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:24.495137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:24.495061    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:24.768107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:24.768027    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:24.928753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:24.928718    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:24.929564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:24.929469    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:24.930042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:24.930026    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:24:25.170137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:25.170105    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:25.638124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:25.638032    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:25.812713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:25.811311    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:26.477713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.476785    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:26.478086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.478055    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:26.786766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.771800    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:26.786766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.772203    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:24:26.786766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.774670    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:26.786766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.775387    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:26.786766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.775645    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:24:26.931841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.931775    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:26.980531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.980486    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:26.981129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.981100    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:24:26.982614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:26.982587    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.035351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.035308    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.135893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.135790    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:24:27.178205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.177784    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 30 19:24:27.303154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.302712    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.491845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.491789    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.492460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.492431    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:24:27.493453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.493432    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:24:27.516867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.516835    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:24:27.783569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.783475    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.784503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.784481    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:24:27.792720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.786773    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.892421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.789792    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:24:27.892421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.801165    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:24:27.892421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.801222    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:24:27.892421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.789833    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:27.935117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.894085    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:24:27.935117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.895527    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.935117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.897596    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:24:27.954129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.937931    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.993715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.957883    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:24:27.993715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.981903    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.993715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.982151    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.993715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.982361    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:24:27.993715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.982417    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:24:27.993715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.983985    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:27.993715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:27.984199    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:24:28.053590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.038107    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:28.053590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.038175    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.091826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.065203    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:24:28.091826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.065315    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.091826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.065674    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:24:28.165997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.165955    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.166665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.166645    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:24:28.503715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.497840    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.503715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.498176    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.874771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.867648    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.931264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.905124    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:28.933190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.905158    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.933190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.905178    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.933190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.905197    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:28.933190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:28.930774    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.002713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.001888    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.055100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.053005    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.055100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.053370    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.055100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.054461    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.073724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.067802    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.174713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.174512    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.216713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.216258    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:29.216713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.216320    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.263715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.245820    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded"
Oct 30 19:24:29.263715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.246934    4925 patch_prober.go:29] interesting pod/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:29.263715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.246982    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.337869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.299525    4925 patch_prober.go:29] interesting pod/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:29.337869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.299593    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.500901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.500060    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.500901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.500465    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.561118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.561082    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:29.561360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.561332    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.934713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.933814    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:29.934713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.934220    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.934713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.934516    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:29.938711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.935746    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.006715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:29.994457    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"]
Oct 30 19:24:30.008582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.008550    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.047715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.025931    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.047715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.035954    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"]
Oct 30 19:24:30.062714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.053785    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.062714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.054185    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.063651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.063049    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.063651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.063392    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.069751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.065991    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:24:30.069751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.066062    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:24:30.069751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.066237    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-642gd\""
Oct 30 19:24:30.069751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.066326    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:24:30.069751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.069584    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:30.069751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.069628    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.069751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.069724    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.089713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.086241    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.089713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.086300    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.089713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.086453    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.089713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.086522    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qh6\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-kube-api-access-26qh6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.089713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.086579    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.089713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.086655    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/94931795-ec51-464e-aeb7-104def6ebcd3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.089713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.086779    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.136206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.109060    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"]
Oct 30 19:24:30.197766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.187973    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.197766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.188079    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/94931795-ec51-464e-aeb7-104def6ebcd3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.197766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.188185    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.197766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.188245    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.197766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.188275    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.197766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.188428    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.197766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.188501    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26qh6\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-kube-api-access-26qh6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.197766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.189055    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.197766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.192252    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.220011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.219286    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.220011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.219950    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/94931795-ec51-464e-aeb7-104def6ebcd3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.247270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.247241    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:24:30.252260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.251811    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:30.311457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.311420    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:30.388714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.325715    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"]
Oct 30 19:24:30.388714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.326827    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d2abb4-4q7wx\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.388714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.375862    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbpx\" (UniqueName: \"kubernetes.io/projected/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-kube-api-access-zzbpx\") pod \"maas-api-revoke-sub-28d2abb4-4q7wx\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.388714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.376164    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d2abb4-4q7wx\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.388714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.373476    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.392818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.389582    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qh6\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-kube-api-access-26qh6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.438645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.438270    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:30.438645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.438338    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.487901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.477748    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d2abb4-4q7wx\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.487901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.477839    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d2abb4-4q7wx\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.487901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.478047    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbpx\" (UniqueName: \"kubernetes.io/projected/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-kube-api-access-zzbpx\") pod \"maas-api-revoke-sub-28d2abb4-4q7wx\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.487901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.479116    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d2abb4-4q7wx\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.511500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.510806    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.511500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.510958    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.511500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.511035    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:24:30.576291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.576258    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.717714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.672458    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d2abb4-4q7wx\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.717999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.714466    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbpx\" (UniqueName: \"kubernetes.io/projected/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-kube-api-access-zzbpx\") pod \"maas-api-revoke-sub-28d2abb4-4q7wx\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.726108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.725981    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.753770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.735962    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:30.767930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.767797    4925 patch_prober.go:29] interesting pod/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:30.767930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.767862    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:30.774361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.747069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:30.774361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.774014    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5j4cl\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:30.831702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.805407    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.070832    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.949026    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.950295    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.950325    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.950368    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:30.859839    4925 patch_prober.go:29] interesting pod/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.073634    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.074642    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.074646    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.075884    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.076496    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.090816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.076759    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:24:31.216744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.201967    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:24:31.223811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.223766    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.633410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.633364    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:31.654750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:31.639171    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:32.140716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.109647    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:24:32.161814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.144953    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:32.162779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.145218    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:32.191986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.145322    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:24:32.507204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.507161    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:32.668093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.541237    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:24:32.668559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.541764    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:24:32.668935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.541928    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:24:32.669201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.645846    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:32.669300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.645887    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:32.669397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.647837    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:24:32.757905 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:24:32.757841    4925 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.489s"
Oct 30 19:24:32.758029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.757952    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:24:32.758029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.757993    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:24:32.758029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.758025    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:24:32.758185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.758055    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:24:32.758185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.758086    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:24:32.768523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.767536    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:24:32.782711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.778409    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:24:32.788102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.788065    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"]
Oct 30 19:24:32.793466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.793446    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:24:32.793579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.793476    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"]
Oct 30 19:24:32.793579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.793521    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:24:32.793579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.793552    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:24:32.793767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.793749    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:32.794127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.794107    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:32.804511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.804489    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:24:32.822739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.822556    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:24:32.920701 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.920651    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26db3c94-8qhzr\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:32.920922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.920900    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26db3c94-8qhzr\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:32.921000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.920965    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp2xx\" (UniqueName: \"kubernetes.io/projected/e815abb7-3ccd-45af-8808-e29d808b3572-kube-api-access-xp2xx\") pod \"maas-api-revoke-sub-26db3c94-8qhzr\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:32.961605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.961562    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:32.978509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.978475    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"]
Oct 30 19:24:32.979615 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:24:32.979586    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94931795_ec51_464e_aeb7_104def6ebcd3.slice/crio-71fae5eaad5168c392b1b85857a13522583a19a7ce2dc692bbda498b39c89d74 WatchSource:0}: Error finding container 71fae5eaad5168c392b1b85857a13522583a19a7ce2dc692bbda498b39c89d74: Status 404 returned error can't find the container with id 71fae5eaad5168c392b1b85857a13522583a19a7ce2dc692bbda498b39c89d74
Oct 30 19:24:32.982441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:32.982420    4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:24:33.029819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.029713    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26db3c94-8qhzr\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:33.029988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.029845    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26db3c94-8qhzr\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:33.029988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.029886    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp2xx\" (UniqueName: \"kubernetes.io/projected/e815abb7-3ccd-45af-8808-e29d808b3572-kube-api-access-xp2xx\") pod \"maas-api-revoke-sub-26db3c94-8qhzr\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:33.031199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.031173    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26db3c94-8qhzr\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:33.034952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.034928    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26db3c94-8qhzr\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:33.050397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.050369    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp2xx\" (UniqueName: \"kubernetes.io/projected/e815abb7-3ccd-45af-8808-e29d808b3572-kube-api-access-xp2xx\") pod \"maas-api-revoke-sub-26db3c94-8qhzr\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:33.080029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.079999    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"]
Oct 30 19:24:33.087907 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:24:33.087877    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2684b4ec_8fdd_4ade_b8fb_fabfc8f4b5cf.slice/crio-86c1ab67c5ce8bf3f0b019340dca95b75d96daa8bd3efb4ed9d32fe8c999d750 WatchSource:0}: Error finding container 86c1ab67c5ce8bf3f0b019340dca95b75d96daa8bd3efb4ed9d32fe8c999d750: Status 404 returned error can't find the container with id 86c1ab67c5ce8bf3f0b019340dca95b75d96daa8bd3efb4ed9d32fe8c999d750
Oct 30 19:24:33.120965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.120945    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:33.414073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.414044    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"]
Oct 30 19:24:33.415020 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:24:33.414982    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode815abb7_3ccd_45af_8808_e29d808b3572.slice/crio-6629582d86eb5aa4eb98191bd77a6d25115c8aeab3ad9f110e578ba3caa9b290 WatchSource:0}: Error finding container 6629582d86eb5aa4eb98191bd77a6d25115c8aeab3ad9f110e578ba3caa9b290: Status 404 returned error can't find the container with id 6629582d86eb5aa4eb98191bd77a6d25115c8aeab3ad9f110e578ba3caa9b290
Oct 30 19:24:33.792634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.792598    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr" event={"ID":"e815abb7-3ccd-45af-8808-e29d808b3572","Type":"ContainerStarted","Data":"e25805366ec6e869a26ebb2484487be0f32c99e76734ce8dee84f195424888f3"}
Oct 30 19:24:33.792853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.792663    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr" event={"ID":"e815abb7-3ccd-45af-8808-e29d808b3572","Type":"ContainerStarted","Data":"6629582d86eb5aa4eb98191bd77a6d25115c8aeab3ad9f110e578ba3caa9b290"}
Oct 30 19:24:33.794601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.794571    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx" event={"ID":"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf","Type":"ContainerStarted","Data":"3c76df8830ccfcbf5fe3fae63c7fe4eeb5db7db7a876f4a7bd4c39fe09e896c2"}
Oct 30 19:24:33.794764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.794627    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx" event={"ID":"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf","Type":"ContainerStarted","Data":"86c1ab67c5ce8bf3f0b019340dca95b75d96daa8bd3efb4ed9d32fe8c999d750"}
Oct 30 19:24:33.796456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.796434    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl" event={"ID":"94931795-ec51-464e-aeb7-104def6ebcd3","Type":"ContainerStarted","Data":"71fae5eaad5168c392b1b85857a13522583a19a7ce2dc692bbda498b39c89d74"}
Oct 30 19:24:33.811767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.811721    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr" podStartSLOduration=2.811673287 podStartE2EDuration="2.811673287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:33.810497304 +0000 UTC m=+1108.713510341" watchObservedRunningTime="2026-10-30 19:24:33.811673287 +0000 UTC m=+1108.714686323"
Oct 30 19:24:33.829317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:33.829271    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx" podStartSLOduration=4.829260253 podStartE2EDuration="4.829260253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:33.826451504 +0000 UTC m=+1108.729464541" watchObservedRunningTime="2026-10-30 19:24:33.829260253 +0000 UTC m=+1108.732273290"
Oct 30 19:24:34.017012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:34.007783    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:34.754590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:34.754479    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:34.809949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:34.809910    4925 generic.go:356] "Generic (PLEG): container finished" podID="2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf" containerID="3c76df8830ccfcbf5fe3fae63c7fe4eeb5db7db7a876f4a7bd4c39fe09e896c2" exitCode=0
Oct 30 19:24:34.810565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:34.810537    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx" event={"ID":"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf","Type":"ContainerDied","Data":"3c76df8830ccfcbf5fe3fae63c7fe4eeb5db7db7a876f4a7bd4c39fe09e896c2"}
Oct 30 19:24:34.812919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:34.812888    4925 generic.go:356] "Generic (PLEG): container finished" podID="e815abb7-3ccd-45af-8808-e29d808b3572" containerID="e25805366ec6e869a26ebb2484487be0f32c99e76734ce8dee84f195424888f3" exitCode=0
Oct 30 19:24:34.813046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:34.812940    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr" event={"ID":"e815abb7-3ccd-45af-8808-e29d808b3572","Type":"ContainerDied","Data":"e25805366ec6e869a26ebb2484487be0f32c99e76734ce8dee84f195424888f3"}
Oct 30 19:24:35.736470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:35.736416    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:35.827960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:35.827909    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:24:35.828262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:35.828004    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:24:35.902652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:35.902062    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:36.077907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.077398    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:36.105729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.104625    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:36.147100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.144069    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:36.209764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.209730    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e815abb7-3ccd-45af-8808-e29d808b3572-kube-api-access-xp2xx\" (UniqueName: \"kubernetes.io/projected/e815abb7-3ccd-45af-8808-e29d808b3572-kube-api-access-xp2xx\") pod \"e815abb7-3ccd-45af-8808-e29d808b3572\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") "
Oct 30 19:24:36.209954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.209791    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-openshift-ca\") pod \"e815abb7-3ccd-45af-8808-e29d808b3572\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") "
Oct 30 19:24:36.209954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.209828    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-kube-api-access-zzbpx\" (UniqueName: \"kubernetes.io/projected/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-kube-api-access-zzbpx\") pod \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") "
Oct 30 19:24:36.209954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.209938    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-openshift-ca\") pod \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") "
Oct 30 19:24:36.211650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.211621    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-openshift-ca" pod "e815abb7-3ccd-45af-8808-e29d808b3572" (UID: "e815abb7-3ccd-45af-8808-e29d808b3572"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:36.215965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.214193    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-xks-ca\") pod \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\" (UID: \"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf\") "
Oct 30 19:24:36.215965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.214383    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-xks-ca\") pod \"e815abb7-3ccd-45af-8808-e29d808b3572\" (UID: \"e815abb7-3ccd-45af-8808-e29d808b3572\") "
Oct 30 19:24:36.215965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.215095    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.215965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.215740    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-openshift-ca" pod "2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf" (UID: "2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:36.225736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.218442    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-xks-ca" pod "e815abb7-3ccd-45af-8808-e29d808b3572" (UID: "e815abb7-3ccd-45af-8808-e29d808b3572"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:36.225736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.218613    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-kube-api-access-zzbpx" pod "2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf" (UID: "2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf"). InnerVolumeSpecName "kube-api-access-zzbpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:36.225736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.218803    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e815abb7-3ccd-45af-8808-e29d808b3572-kube-api-access-xp2xx" pod "e815abb7-3ccd-45af-8808-e29d808b3572" (UID: "e815abb7-3ccd-45af-8808-e29d808b3572"). InnerVolumeSpecName "kube-api-access-xp2xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:36.225736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.221875    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-xks-ca" pod "2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf" (UID: "2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:36.318955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.317106    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815abb7-3ccd-45af-8808-e29d808b3572-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.318955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.317143    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp2xx\" (UniqueName: \"kubernetes.io/projected/e815abb7-3ccd-45af-8808-e29d808b3572-kube-api-access-xp2xx\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.318955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.317159    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzbpx\" (UniqueName: \"kubernetes.io/projected/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-kube-api-access-zzbpx\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.318955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.317173    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.318955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.317186    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:36.830349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.830035    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:36.837764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.836748    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"
Oct 30 19:24:36.839873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.839834    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr" event={"ID":"e815abb7-3ccd-45af-8808-e29d808b3572","Type":"ContainerDied","Data":"6629582d86eb5aa4eb98191bd77a6d25115c8aeab3ad9f110e578ba3caa9b290"}
Oct 30 19:24:36.840032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.839902    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6629582d86eb5aa4eb98191bd77a6d25115c8aeab3ad9f110e578ba3caa9b290"
Oct 30 19:24:36.841744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.841713    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx" event={"ID":"2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf","Type":"ContainerDied","Data":"86c1ab67c5ce8bf3f0b019340dca95b75d96daa8bd3efb4ed9d32fe8c999d750"}
Oct 30 19:24:36.841883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.841768    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86c1ab67c5ce8bf3f0b019340dca95b75d96daa8bd3efb4ed9d32fe8c999d750"
Oct 30 19:24:36.841883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.841677    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"
Oct 30 19:24:36.846396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.846366    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl" event={"ID":"94931795-ec51-464e-aeb7-104def6ebcd3","Type":"ContainerStarted","Data":"4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e"}
Oct 30 19:24:36.846611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.846584    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:36.850198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.850169    4925 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5j4cl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:24:36.850311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.850215    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl" podUID="94931795-ec51-464e-aeb7-104def6ebcd3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:24:36.876942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:36.876890    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl" podStartSLOduration=6.033370374 podStartE2EDuration="8.876873803s" totalImagesPullingTime="2.843503429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:36.869645788 +0000 UTC m=+1111.772658835" watchObservedRunningTime="2026-10-30 19:24:36.876873803 +0000 UTC m=+1111.779886846"
Oct 30 19:24:37.757850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:37.757801    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:37.854260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:37.854231    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:24:38.472083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:38.472029    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:39.422398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:39.422366    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:39.618336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:39.618304    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:39.943596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:39.943566    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:40.116112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:40.116081    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:40.530551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:40.530518    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:40.702237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:40.702202    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:40.878297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:40.878201    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:41.513971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:41.513937    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:41.743010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:41.742966    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:42.655002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:42.654971    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:43.436993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:43.436901    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:43.993883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:43.993849    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:44.156259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:44.156218    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:44.453267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:44.453229    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:44.620550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:44.620519    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:45.074987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:45.074954    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:45.517419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:45.517382    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:46.344203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:46.344124    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:47.302041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:47.302007    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:47.880727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:47.880663    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:48.127000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:48.126964    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:48.351256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:48.351211    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:49.089815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:49.089781    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:49.301041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:49.288373    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:50.224731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:50.224678    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:51.170751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:51.170671    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:52.144468 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:24:52.144428    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:24:52.147491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:52.147462    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:53.798485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:53.798453    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:54.225827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:54.225782    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.503770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:54.502780    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.528318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:54.528270    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:24:54.572838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:54.572789    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.672712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:54.668807    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:24:54.941860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:54.941803    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:54.942863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:54.942833    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:54.943307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:54.943284    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:24:54.947251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:54.947210    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.009907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.009847    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.112749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.085781    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.112749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.087331    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.220882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.220787    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:24:55.435399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.221056    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Oct 30 19:24:55.439983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.222976    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.459704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.333977    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:24:55.521805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.517928    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:24:55.534718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.528662    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.695742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.694422    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.810622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.810536    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:24:55.811171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.811154    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:24:55.812192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.812177    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:24:55.838513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.838488    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:24:55.894436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.894405    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:55.919459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.919132    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:24:55.919774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.919746    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.920148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.919266    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:55.955222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.955181    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:56.039745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.039429    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:24:56.040416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.957861    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:24:56.073768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.966950    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:24:56.073768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.062029    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:56.073768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:55.966913    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:24:56.104634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.104595    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:24:56.105378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.105359    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:24:56.134745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.108582    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:56.134745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.108615    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:24:56.175756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.135302    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:24:56.200713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.176163    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:24:56.270633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.201300    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:24:56.288712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.259504    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:56.289046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.259559    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:56.290228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.289742    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:24:56.333768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.330242    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:56.462611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.462571    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:24:56.477714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.464544    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:24:56.507734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.505032    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:56.507734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.505759    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:24:56.507734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.506376    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:24:56.596403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.596301    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:56.597031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.596995    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:24:56.615181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.615104    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:56.716011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.709229    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:56.793224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.793193    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:24:56.995789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.995748    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:57.034065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:56.996289    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:24:57.076469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.069952    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:57.076469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.073855    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:57.082704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.079816    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:57.082704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.080459    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:24:57.085233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.084904    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:24:57.187466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.186845    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:57.490056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.490013    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:57.587282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.574310    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:24:57.587282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.574534    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:57.664770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.630591    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:57.701729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.692192    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:24:57.743740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.731224    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:57.822434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.822389    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:57.852305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:57.851305    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:58.013811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.009352    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:58.114770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.114596    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:58.144897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.144856    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:58.175164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.175119    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:24:58.191614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.191584    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:24:58.191919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.191902    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:24:58.246996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.246968    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:24:58.280211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.280131    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:24:58.280724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.280673    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:24:58.383732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.381579    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:24:58.392921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.392857    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:24:58.392921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.392918    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:24:58.466645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.466621    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:24:58.473145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.473120    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:24:58.684030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:58.683909    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:59.561738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:59.560194    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:59.733902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:24:59.733871    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:00.672865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:00.672822    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:01.557211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:01.557184    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:02.187462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:02.187427    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:02.968800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:02.968765    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:03.377765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:03.377652    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:03.655759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:03.655448    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:04.363011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:04.362976    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:04.854757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:04.854728    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:05.021123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:05.021075    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:05.182139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:05.182078    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:05.436451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:05.424563    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:05.607274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:05.607074    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:05.949056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:05.948639    4925 patch_prober.go:29] interesting pod/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": read tcp 10.132.0.2:36750->10.132.0.100:15021: read: connection reset by peer" start-of-body=
Oct 30 19:25:05.949056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:05.948737    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh" podUID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": read tcp 10.132.0.2:36750->10.132.0.100:15021: read: connection reset by peer"
Oct 30 19:25:06.039378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.039341    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:06.223358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.223258    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:06.279456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.279410    4925 generic.go:356] "Generic (PLEG): container finished" podID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerID="6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9" exitCode=0
Oct 30 19:25:06.279658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.279516    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" event={"ID":"f711c427-67a2-4e2c-8d39-017ae8b90ac6","Type":"ContainerDied","Data":"6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9"}
Oct 30 19:25:06.280162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.280142    4925 scope.go:122] "RemoveContainer" containerID="6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9"
Oct 30 19:25:06.282127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.282079    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:06.282282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.282259    4925 generic.go:356] "Generic (PLEG): container finished" podID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerID="47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a" exitCode=0
Oct 30 19:25:06.282427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.282332    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:06.282427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.282339    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh" event={"ID":"486ee0c9-a8fd-41b5-9d86-e521411aa661","Type":"ContainerDied","Data":"47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a"}
Oct 30 19:25:06.282427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.282388    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:06.282831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.282809    4925 scope.go:122] "RemoveContainer" containerID="47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a"
Oct 30 19:25:06.284404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.284367    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:06.284477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.284464    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:06.284523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.284513    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:06.757608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.757578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:06.932712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:06.931826    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:07.097766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:07.097260    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"]
Oct 30 19:25:07.208710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:07.197554    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:07.524990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:07.524861    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:07.579007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:07.578973    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" event={"ID":"f711c427-67a2-4e2c-8d39-017ae8b90ac6","Type":"ContainerStarted","Data":"dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb"}
Oct 30 19:25:07.591487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:07.591446    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh" event={"ID":"486ee0c9-a8fd-41b5-9d86-e521411aa661","Type":"ContainerStarted","Data":"01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188"}
Oct 30 19:25:07.591708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:07.591654    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh" podUID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerName="istio-proxy" containerID="cri-o://01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188" gracePeriod=30
Oct 30 19:25:07.834131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:07.823028    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:07.991785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:07.991760    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:08.268797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:08.268767    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:25:08.269509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:08.269486    4925 patch_prober.go:29] interesting pod/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:25:08.270193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:08.269669    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:25:08.395714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:08.389215    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:08.555508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:08.554450    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:08.840288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:08.840197    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:09.249708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:09.249086    4925 patch_prober.go:29] interesting pod/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:25:09.249708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:09.249159    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:25:09.292715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:09.285240    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:09.627772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:09.627220    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:09.932708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:09.929193    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:25:10.239767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:10.239060    4925 patch_prober.go:29] interesting pod/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:25:10.239767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:10.239132    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:25:10.742717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:10.737716    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:11.237724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:11.237062    4925 patch_prober.go:29] interesting pod/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:25:11.237724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:11.237143    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:25:11.274665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:11.274643    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:11.476712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:11.476559    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:11.779711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:11.775582    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:12.071787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:12.050353    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:12.247331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:12.241030    4925 patch_prober.go:29] interesting pod/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:25:12.247331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:12.241091    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:25:12.255299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:12.253264    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:12.277966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:12.277923    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"]
Oct 30 19:25:12.278514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:12.278434    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy" containerID="cri-o://dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb" gracePeriod=30
Oct 30 19:25:12.563473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:12.563441    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:12.897581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:12.897549    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:13.316720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:13.316599    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:13.496709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:13.491261    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:14.564632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:14.564588    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:15.164397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.164362    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"]
Oct 30 19:25:15.164985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.164970    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf" containerName="revoke-keys"
Oct 30 19:25:15.165051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.164985    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf" containerName="revoke-keys"
Oct 30 19:25:15.165051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.165034    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e815abb7-3ccd-45af-8808-e29d808b3572" containerName="revoke-keys"
Oct 30 19:25:15.165051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.165040    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e815abb7-3ccd-45af-8808-e29d808b3572" containerName="revoke-keys"
Oct 30 19:25:15.165202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.165146    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e815abb7-3ccd-45af-8808-e29d808b3572" containerName="revoke-keys"
Oct 30 19:25:15.165202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.165158    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf" containerName="revoke-keys"
Oct 30 19:25:15.169839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.169820    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.177215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.177189    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"]
Oct 30 19:25:15.178250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.178171    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:25:15.178250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.178213    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:15.241208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.241178    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c1c8043-kwf77\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.241357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.241227    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c1c8043-kwf77\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.241357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.241343    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfkj\" (UniqueName: \"kubernetes.io/projected/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-kube-api-access-cxfkj\") pod \"maas-api-revoke-sub-2c1c8043-kwf77\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.342343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.342308    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c1c8043-kwf77\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.342343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.342348    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c1c8043-kwf77\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.342552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.342380    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfkj\" (UniqueName: \"kubernetes.io/projected/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-kube-api-access-cxfkj\") pod \"maas-api-revoke-sub-2c1c8043-kwf77\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.343062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.342993    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c1c8043-kwf77\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.344914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.344891    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c1c8043-kwf77\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.354086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.354065    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfkj\" (UniqueName: \"kubernetes.io/projected/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-kube-api-access-cxfkj\") pod \"maas-api-revoke-sub-2c1c8043-kwf77\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.455104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.454940    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:15.490043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.490010    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:15.796540 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:25:15.796507    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03ac7df_c750_4d22_a756_0a0ee2e23d7d.slice/crio-8b4f54aea86f651c3033fa2740635df42479e708225e7236ec3f518dbcb76ed2 WatchSource:0}: Error finding container 8b4f54aea86f651c3033fa2740635df42479e708225e7236ec3f518dbcb76ed2: Status 404 returned error can't find the container with id 8b4f54aea86f651c3033fa2740635df42479e708225e7236ec3f518dbcb76ed2
Oct 30 19:25:15.797041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:15.796853    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"]
Oct 30 19:25:16.048560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:16.048479    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:16.093284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:16.093241    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77" event={"ID":"b03ac7df-c750-4d22-a756-0a0ee2e23d7d","Type":"ContainerStarted","Data":"8b4f54aea86f651c3033fa2740635df42479e708225e7236ec3f518dbcb76ed2"}
Oct 30 19:25:16.210044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:16.209995    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:16.428168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:16.428138    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:16.855515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:16.855371    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:17.104116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:17.104038    4925 generic.go:356] "Generic (PLEG): container finished" podID="b03ac7df-c750-4d22-a756-0a0ee2e23d7d" containerID="061afec097e395fa7bfba1410bf9862d70766d437ba2629dbc9697da9b54440a" exitCode=0
Oct 30 19:25:17.104116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:17.104099    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77" event={"ID":"b03ac7df-c750-4d22-a756-0a0ee2e23d7d","Type":"ContainerDied","Data":"061afec097e395fa7bfba1410bf9862d70766d437ba2629dbc9697da9b54440a"}
Oct 30 19:25:17.733085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:17.732438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:18.289360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.289327    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:18.365410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.365376    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:18.397017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.396988    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-openshift-ca\") pod \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") "
Oct 30 19:25:18.397412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.397390    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-openshift-ca" pod "b03ac7df-c750-4d22-a756-0a0ee2e23d7d" (UID: "b03ac7df-c750-4d22-a756-0a0ee2e23d7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:18.397564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.397417    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-xks-ca\") pod \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") "
Oct 30 19:25:18.397799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.397780    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-kube-api-access-cxfkj\" (UniqueName: \"kubernetes.io/projected/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-kube-api-access-cxfkj\") pod \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\" (UID: \"b03ac7df-c750-4d22-a756-0a0ee2e23d7d\") "
Oct 30 19:25:18.398626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.398602    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.400799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.400770    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-kube-api-access-cxfkj" pod "b03ac7df-c750-4d22-a756-0a0ee2e23d7d" (UID: "b03ac7df-c750-4d22-a756-0a0ee2e23d7d"). InnerVolumeSpecName "kube-api-access-cxfkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:18.402267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.402245    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-xks-ca" pod "b03ac7df-c750-4d22-a756-0a0ee2e23d7d" (UID: "b03ac7df-c750-4d22-a756-0a0ee2e23d7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:18.500824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.500791    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxfkj\" (UniqueName: \"kubernetes.io/projected/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-kube-api-access-cxfkj\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.500824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.500816    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03ac7df-c750-4d22-a756-0a0ee2e23d7d-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.973424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:18.973389    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:19.128221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.128191    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"
Oct 30 19:25:19.128436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.128227    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77" event={"ID":"b03ac7df-c750-4d22-a756-0a0ee2e23d7d","Type":"ContainerDied","Data":"8b4f54aea86f651c3033fa2740635df42479e708225e7236ec3f518dbcb76ed2"}
Oct 30 19:25:19.128436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.128348    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b4f54aea86f651c3033fa2740635df42479e708225e7236ec3f518dbcb76ed2"
Oct 30 19:25:19.133284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.133259    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:19.292399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.292371    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:19.446064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.446032    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:25:19.539300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.539226    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istiod-ca-cert\") pod \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") "
Oct 30 19:25:19.539619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.539596    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istiod-ca-cert" pod "f711c427-67a2-4e2c-8d39-017ae8b90ac6" (UID: "f711c427-67a2-4e2c-8d39-017ae8b90ac6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:19.540005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.539986    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-certs" pod "f711c427-67a2-4e2c-8d39-017ae8b90ac6" (UID: "f711c427-67a2-4e2c-8d39-017ae8b90ac6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.540070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.540032    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-certs\") pod \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") "
Oct 30 19:25:19.540103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.540090    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-credential-socket\") pod \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") "
Oct 30 19:25:19.540160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.540145    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-podinfo\") pod \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") "
Oct 30 19:25:19.540198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.540187    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-socket\") pod \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") "
Oct 30 19:25:19.540231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.540214    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-data\") pod \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") "
Oct 30 19:25:19.540264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.540252    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-kube-api-access-8h46g\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-kube-api-access-8h46g\") pod \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") "
Oct 30 19:25:19.540375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.540362    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-envoy\") pod \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") "
Oct 30 19:25:19.540530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.540515    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-token\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-token\") pod \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\" (UID: \"f711c427-67a2-4e2c-8d39-017ae8b90ac6\") "
Oct 30 19:25:19.541257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.541240    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.541303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.541265    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.541775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.541750    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-socket" pod "f711c427-67a2-4e2c-8d39-017ae8b90ac6" (UID: "f711c427-67a2-4e2c-8d39-017ae8b90ac6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.542124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.542104    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-credential-socket" pod "f711c427-67a2-4e2c-8d39-017ae8b90ac6" (UID: "f711c427-67a2-4e2c-8d39-017ae8b90ac6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.542419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.542117    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-data" pod "f711c427-67a2-4e2c-8d39-017ae8b90ac6" (UID: "f711c427-67a2-4e2c-8d39-017ae8b90ac6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.544958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.544913    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-podinfo" pod "f711c427-67a2-4e2c-8d39-017ae8b90ac6" (UID: "f711c427-67a2-4e2c-8d39-017ae8b90ac6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:25:19.546700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.546658    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-kube-api-access-8h46g" pod "f711c427-67a2-4e2c-8d39-017ae8b90ac6" (UID: "f711c427-67a2-4e2c-8d39-017ae8b90ac6"). InnerVolumeSpecName "kube-api-access-8h46g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:19.554277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.554245    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-envoy" pod "f711c427-67a2-4e2c-8d39-017ae8b90ac6" (UID: "f711c427-67a2-4e2c-8d39-017ae8b90ac6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.554511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.554483    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-token" pod "f711c427-67a2-4e2c-8d39-017ae8b90ac6" (UID: "f711c427-67a2-4e2c-8d39-017ae8b90ac6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:19.587309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.587287    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.642176    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-credential-socket\") pod \"486ee0c9-a8fd-41b5-9d86-e521411aa661\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") "
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.642251    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-envoy\") pod \"486ee0c9-a8fd-41b5-9d86-e521411aa661\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") "
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.642306    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-kube-api-access-thccp\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-kube-api-access-thccp\") pod \"486ee0c9-a8fd-41b5-9d86-e521411aa661\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") "
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.642458    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-certs\") pod \"486ee0c9-a8fd-41b5-9d86-e521411aa661\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") "
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.642520    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-token\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-token\") pod \"486ee0c9-a8fd-41b5-9d86-e521411aa661\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") "
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.642564    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-podinfo\") pod \"486ee0c9-a8fd-41b5-9d86-e521411aa661\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") "
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.642612    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486ee0c9-a8fd-41b5-9d86-e521411aa661-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486ee0c9-a8fd-41b5-9d86-e521411aa661-istiod-ca-cert\") pod \"486ee0c9-a8fd-41b5-9d86-e521411aa661\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") "
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.642704    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-socket\") pod \"486ee0c9-a8fd-41b5-9d86-e521411aa661\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") "
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.642803    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-data\") pod \"486ee0c9-a8fd-41b5-9d86-e521411aa661\" (UID: \"486ee0c9-a8fd-41b5-9d86-e521411aa661\") "
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.643479    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.643503    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.643518    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.643532    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.643547    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.643560    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f711c427-67a2-4e2c-8d39-017ae8b90ac6-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.643575    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h46g\" (UniqueName: \"kubernetes.io/projected/f711c427-67a2-4e2c-8d39-017ae8b90ac6-kube-api-access-8h46g\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.644005    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-certs" pod "486ee0c9-a8fd-41b5-9d86-e521411aa661" (UID: "486ee0c9-a8fd-41b5-9d86-e521411aa661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.645768    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-data" pod "486ee0c9-a8fd-41b5-9d86-e521411aa661" (UID: "486ee0c9-a8fd-41b5-9d86-e521411aa661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.646210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.646114    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-credential-socket" pod "486ee0c9-a8fd-41b5-9d86-e521411aa661" (UID: "486ee0c9-a8fd-41b5-9d86-e521411aa661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.649275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.648999    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-envoy" pod "486ee0c9-a8fd-41b5-9d86-e521411aa661" (UID: "486ee0c9-a8fd-41b5-9d86-e521411aa661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.657929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.653131    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-token" pod "486ee0c9-a8fd-41b5-9d86-e521411aa661" (UID: "486ee0c9-a8fd-41b5-9d86-e521411aa661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:19.657929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.653327    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-socket" pod "486ee0c9-a8fd-41b5-9d86-e521411aa661" (UID: "486ee0c9-a8fd-41b5-9d86-e521411aa661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:19.657929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.654522    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486ee0c9-a8fd-41b5-9d86-e521411aa661-istiod-ca-cert" pod "486ee0c9-a8fd-41b5-9d86-e521411aa661" (UID: "486ee0c9-a8fd-41b5-9d86-e521411aa661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:19.657929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.656043    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-kube-api-access-thccp" pod "486ee0c9-a8fd-41b5-9d86-e521411aa661" (UID: "486ee0c9-a8fd-41b5-9d86-e521411aa661"). InnerVolumeSpecName "kube-api-access-thccp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:19.657929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.657559    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-podinfo" pod "486ee0c9-a8fd-41b5-9d86-e521411aa661" (UID: "486ee0c9-a8fd-41b5-9d86-e521411aa661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:25:19.745306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.745275    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.745545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.745528    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.745658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.745643    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486ee0c9-a8fd-41b5-9d86-e521411aa661-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.745812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.745798    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.745915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.745903    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.746005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.745993    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.746091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.746081    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.746176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.746165    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thccp\" (UniqueName: \"kubernetes.io/projected/486ee0c9-a8fd-41b5-9d86-e521411aa661-kube-api-access-thccp\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:19.746265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:19.746253    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486ee0c9-a8fd-41b5-9d86-e521411aa661-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:20.106430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.106393    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:20.137011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.136979    4925 generic.go:356] "Generic (PLEG): container finished" podID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerID="dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb" exitCode=0
Oct 30 19:25:20.137205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.137038    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"
Oct 30 19:25:20.137205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.137056    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" event={"ID":"f711c427-67a2-4e2c-8d39-017ae8b90ac6","Type":"ContainerDied","Data":"dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb"}
Oct 30 19:25:20.137205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.137118    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282" event={"ID":"f711c427-67a2-4e2c-8d39-017ae8b90ac6","Type":"ContainerDied","Data":"924d82f9db79150293de83c7e175b5071fde98370b06ea7f4a8f74aa3130a171"}
Oct 30 19:25:20.137205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.137144    4925 scope.go:122] "RemoveContainer" containerID="dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb"
Oct 30 19:25:20.139419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.139391    4925 generic.go:356] "Generic (PLEG): container finished" podID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerID="01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188" exitCode=0
Oct 30 19:25:20.139593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.139443    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"
Oct 30 19:25:20.139593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.139479    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh" event={"ID":"486ee0c9-a8fd-41b5-9d86-e521411aa661","Type":"ContainerDied","Data":"01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188"}
Oct 30 19:25:20.139593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.139555    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh" event={"ID":"486ee0c9-a8fd-41b5-9d86-e521411aa661","Type":"ContainerDied","Data":"e2c15f5963bb2f3321e7aa3d78867fffe17d6718f01589cb19d098b9bd55908d"}
Oct 30 19:25:20.150314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.150298    4925 scope.go:122] "RemoveContainer" containerID="6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9"
Oct 30 19:25:20.161936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.161916    4925 scope.go:122] "RemoveContainer" containerID="dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb"
Oct 30 19:25:20.162331 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:25:20.162305    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb\": container with ID starting with dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb not found: ID does not exist" containerID="dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb"
Oct 30 19:25:20.162425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.162341    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb"} err="failed to get container status \"dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb\": rpc error: code = NotFound desc = could not find container \"dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb\": container with ID starting with dba3c370171beb9635c3ef2ce39004954f042d525690cb1a4229047bb90de4bb not found: ID does not exist"
Oct 30 19:25:20.162425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.162361    4925 scope.go:122] "RemoveContainer" containerID="6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9"
Oct 30 19:25:20.162753 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:25:20.162726    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9\": container with ID starting with 6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9 not found: ID does not exist" containerID="6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9"
Oct 30 19:25:20.162835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.162761    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9"} err="failed to get container status \"6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9\": rpc error: code = NotFound desc = could not find container \"6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9\": container with ID starting with 6948c7685b2bd424a6fa5928e862f43e3913bc3c4feb0f2b67e4638f1be0b6d9 not found: ID does not exist"
Oct 30 19:25:20.162835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.162784    4925 scope.go:122] "RemoveContainer" containerID="01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188"
Oct 30 19:25:20.175460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.175406    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"]
Oct 30 19:25:20.176194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.176168    4925 scope.go:122] "RemoveContainer" containerID="47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a"
Oct 30 19:25:20.183439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.183415    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-250fb057-openshift-default-7c6cb4fd67-xzthh"]
Oct 30 19:25:20.189284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.189264    4925 scope.go:122] "RemoveContainer" containerID="01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188"
Oct 30 19:25:20.189590 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:25:20.189569    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188\": container with ID starting with 01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188 not found: ID does not exist" containerID="01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188"
Oct 30 19:25:20.189749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.189601    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188"} err="failed to get container status \"01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188\": rpc error: code = NotFound desc = could not find container \"01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188\": container with ID starting with 01ab097307b9cbb77f5fec81560dc327ae4bcf68d98a46751b2e3e78cf827188 not found: ID does not exist"
Oct 30 19:25:20.189749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.189626    4925 scope.go:122] "RemoveContainer" containerID="47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a"
Oct 30 19:25:20.189985 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:25:20.189963    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a\": container with ID starting with 47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a not found: ID does not exist" containerID="47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a"
Oct 30 19:25:20.190073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.189993    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a"} err="failed to get container status \"47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a\": rpc error: code = NotFound desc = could not find container \"47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a\": container with ID starting with 47ab507ce0e41985efa70f91f3f95105339e291eb4d32ced314997ccd2141e8a not found: ID does not exist"
Oct 30 19:25:20.197180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.197156    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"]
Oct 30 19:25:20.204354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.204332    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e308e736-gw-openshift-default-99b946c6f-5w282"]
Oct 30 19:25:20.406138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.406105    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"]
Oct 30 19:25:20.407018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.406993    4925 cpu_manager.go:414] "containerMap: removing container" podUID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerName="istio-proxy"
Oct 30 19:25:20.407018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407017    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerName="istio-proxy"
Oct 30 19:25:20.407222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407067    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b03ac7df-c750-4d22-a756-0a0ee2e23d7d" containerName="revoke-keys"
Oct 30 19:25:20.407222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407075    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03ac7df-c750-4d22-a756-0a0ee2e23d7d" containerName="revoke-keys"
Oct 30 19:25:20.407222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407108    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy"
Oct 30 19:25:20.407222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407115    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy"
Oct 30 19:25:20.407222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407134    4925 cpu_manager.go:414] "containerMap: removing container" podUID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerName="istio-proxy"
Oct 30 19:25:20.407222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407141    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerName="istio-proxy"
Oct 30 19:25:20.407222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407152    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy"
Oct 30 19:25:20.407222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407161    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy"
Oct 30 19:25:20.407609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407326    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerName="istio-proxy"
Oct 30 19:25:20.407609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407345    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03ac7df-c750-4d22-a756-0a0ee2e23d7d" containerName="revoke-keys"
Oct 30 19:25:20.407609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407359    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy"
Oct 30 19:25:20.407609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407369    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="486ee0c9-a8fd-41b5-9d86-e521411aa661" containerName="istio-proxy"
Oct 30 19:25:20.407609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.407380    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" containerName="istio-proxy"
Oct 30 19:25:20.437761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.431975    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"]
Oct 30 19:25:20.437761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.432066    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.558736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.558670    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.558922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.558793    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.558922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.558827    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.559022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.558917    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.559022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.558972    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.559022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.559017    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.559146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.559034    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.559146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.559055    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6np5b\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-kube-api-access-6np5b\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.559146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.559130    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.660656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.660572    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.660656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.660602    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6np5b\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-kube-api-access-6np5b\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.660656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.660636    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.660843    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.660885    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.660910    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.660937    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.660984    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.661046    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.661193    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-credential-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.661281    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-data\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.661321    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-certs\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.661382    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-socket\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.661462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.661439    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istiod-ca-cert\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.663428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.663404    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-envoy\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.663560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.663541    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-podinfo\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.672097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.672067    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-token\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.672299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.672280    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6np5b\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-kube-api-access-6np5b\") pod \"e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") " pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.755811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.755789    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:20.983816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:20.983755    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:21.217037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:21.214996    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:21.217037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:21.215096    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:21.217037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:21.215142    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:21.217293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:21.217228    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"]
Oct 30 19:25:21.290207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:21.290173    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486ee0c9-a8fd-41b5-9d86-e521411aa661" path="/var/lib/kubelet/pods/486ee0c9-a8fd-41b5-9d86-e521411aa661/volumes"
Oct 30 19:25:21.290977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:21.290942    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f711c427-67a2-4e2c-8d39-017ae8b90ac6" path="/var/lib/kubelet/pods/f711c427-67a2-4e2c-8d39-017ae8b90ac6/volumes"
Oct 30 19:25:21.980309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:21.980280    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:22.260769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:22.256878    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" event={"ID":"8999c9ff-9ed0-4705-93e5-6bca3a32ed68","Type":"ContainerStarted","Data":"9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856"}
Oct 30 19:25:22.260769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:22.257061    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" event={"ID":"8999c9ff-9ed0-4705-93e5-6bca3a32ed68","Type":"ContainerStarted","Data":"cdf08ff0880ff550e4c8655f1a9a1707d58c938bdf076bc4136f49e5b8262e14"}
Oct 30 19:25:22.326861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:22.326802    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" podStartSLOduration=2.32678195 podStartE2EDuration="2.32678195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:22.317187629 +0000 UTC m=+1157.220200669" watchObservedRunningTime="2026-10-30 19:25:22.32678195 +0000 UTC m=+1157.229795008"
Oct 30 19:25:22.402986 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:25:22.402933    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:25:22.757727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:22.756412    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:22.765736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:22.759804    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:22.765736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:22.759865    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:25:23.099770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:23.095755    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:23.509716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:23.508774    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:25:23.756716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:23.756157    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:23.756716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:23.756215    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:25:24.681713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.680999    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.682196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.682141    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.773719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.730869    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:25:24.773719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.744098    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:25:24.773719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.744864    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:25:24.773719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.745523    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:25:24.773719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.753848    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:24.820578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.782905    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:25:24.820578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.802484    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:24.820578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:24.802565    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:25:25.019809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.014115    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.082826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.080098    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.136717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.132873    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:25:25.231366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.227955    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.231366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.229378    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:25:25.231366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.229513    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:25:25.414767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.385642    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:25.446022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.445962    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Oct 30 19:25:25.630715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.627948    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.684760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.683781    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.746714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.745791    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.746714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.746053    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.746714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.746390    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:25:25.754428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.754385    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:25.807785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.800020    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:25.807785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.800354    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:25.807785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.800406    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:25:25.807785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.801793    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:25:25.927713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.920665    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:25:25.927713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.920747    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:25:25.927713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.920791    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:25:25.927713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.920825    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:25:25.927713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.920857    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:25:26.000769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:25.995058    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:26.018363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.018323    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:26.094653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.090095    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:26.094653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.090649    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:25:26.145941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.144447    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.145941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.144991    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:25:26.145941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.145889    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:26.146235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.146209    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:25:26.189285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.189249    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:26.280918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.280664    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.281110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.281072    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.281656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.281632    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:25:26.281900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.281883    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:25:26.572044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.542179    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:26.572044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.569127    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:25:26.573252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.573227    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:25:26.760177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.760145    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:26.762803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.762761    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:25:26.779827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.779786    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:26.780172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.780147    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:25:26.785708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.781072    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:26.785708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.781092    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.785708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.781368    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.794379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.794359    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:25:26.806708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.802566    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:25:26.806708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.803716    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:26.819713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.819091    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:25:26.866937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.866839    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:25:26.870190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.870156    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:25:26.880732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.880656    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:25:26.962242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.962201    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:25:26.962853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.962834    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:25:26.972897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.966610    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:25:26.972897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.968123    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:25:26.973595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.973580    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:25:26.989155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:26.989138    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:25:27.018225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:27.017392    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:25:27.129796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:27.120869    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:27.756309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:27.756280    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:27.756482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:27.756329    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:25:28.366123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:28.366068    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:28.756532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:28.756502    4925 patch_prober.go:29] interesting pod/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:25:28.756742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:28.756559    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:25:29.226572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:29.225619    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:29.868165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:29.868101    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:30.036200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:30.036165    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:30.208921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:30.208878    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:30.279221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:30.279196    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:30.279826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:30.279796    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:30.281475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:30.281450    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:25:30.729208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:30.729169    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:31.735614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:31.735578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:31.907746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:31.905729    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:32.728502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:32.728439    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:32.903042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:32.903006    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:33.374204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:33.374170    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:33.700913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:33.700871    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:33.861524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:33.861494    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:34.667979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:34.667947    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:35.856529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:35.856486    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:36.482709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:36.482659    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:36.723570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:36.723537    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:36.896984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:36.896953    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:37.552651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:37.551673    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:38.301924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:38.301866    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:39.045279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:39.045241    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:41.055172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:41.055141    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:41.220362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:41.220329    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:41.643278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:41.643244    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:42.060640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:42.060545    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:42.222715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:42.222271    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:42.558160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:42.558121    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:42.886256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:42.886152    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:43.485260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:43.485217    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:44.321430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:44.321397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:45.340031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:45.339990    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:45.987433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:45.987403    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:46.194823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:46.194791    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:46.387441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:46.387350    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:46.634962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:46.634935    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:46.819764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:46.819735    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:47.301047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:47.301016    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:47.494955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:47.493529    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:47.801665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:47.801632    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:47.962896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:47.962869    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:48.174335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:48.174302    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:48.344208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:48.344168    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:49.621186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:49.621152    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:50.344451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.344413    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:50.510644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.510607    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:50.744225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.744189    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"]
Oct 30 19:25:50.750657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.750635    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:50.754513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.754481    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-618d93-openshift-default-dockercfg-r8kgs\""
Oct 30 19:25:50.770384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.770355    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"]
Oct 30 19:25:50.853718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.849703    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:50.926710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.916520    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-certs\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:50.926710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.916668    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-data\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:50.926710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.916856    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-envoy\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:50.926710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.916950    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-credential-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:50.926710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.917002    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-token\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:50.926710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.917031    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-podinfo\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:50.926710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.917055    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4cpf\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-kube-api-access-j4cpf\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:50.926710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.917091    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f570b-588f-4e4d-839f-423a7fa75cdf-istiod-ca-cert\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:50.926710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:50.917173    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.019144    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.019220    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-certs\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.019288    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-data\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.019427    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-envoy\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.019505    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-credential-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.019547    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-token\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.019573    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-podinfo\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.019597    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4cpf\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-kube-api-access-j4cpf\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.019627    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f570b-588f-4e4d-839f-423a7fa75cdf-istiod-ca-cert\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.020396    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f570b-588f-4e4d-839f-423a7fa75cdf-istiod-ca-cert\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.020754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.020639    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.021482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.020880    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-certs\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.021482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.021073    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-data\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.022212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.022188    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-credential-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.026322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.026273    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-envoy\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.032235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.032209    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-podinfo\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.037439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.037393    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-token\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.040224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.040182    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4cpf\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-kube-api-access-j4cpf\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.056657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.056416    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"]
Oct 30 19:25:51.063865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.063835    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"]
Oct 30 19:25:51.069074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.069045    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:51.070994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.070973    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4aea515-vkl82"]
Oct 30 19:25:51.075386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.075364    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6cde10f-jngdj"]
Oct 30 19:25:51.269441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.269400    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0a53ab-a863-4c36-b396-1ae726360bef" path="/var/lib/kubelet/pods/2c0a53ab-a863-4c36-b396-1ae726360bef/volumes"
Oct 30 19:25:51.272238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.272160    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7962588-dffa-473a-a502-6c26ee508933" path="/var/lib/kubelet/pods/b7962588-dffa-473a-a502-6c26ee508933/volumes"
Oct 30 19:25:51.273467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.273440    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:51.396346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.396315    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"]
Oct 30 19:25:51.396844 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:25:51.396814    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00f570b_588f_4e4d_839f_423a7fa75cdf.slice/crio-20f3d355f98f3cc1a9db5e16ed2f977a50ab218735cb5f42d514be6c59335c1f WatchSource:0}: Error finding container 20f3d355f98f3cc1a9db5e16ed2f977a50ab218735cb5f42d514be6c59335c1f: Status 404 returned error can't find the container with id 20f3d355f98f3cc1a9db5e16ed2f977a50ab218735cb5f42d514be6c59335c1f
Oct 30 19:25:51.399311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.399277    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:51.399422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.399339    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:51.399422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.399369    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:25:51.638870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:51.638788    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:52.117337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:52.117302    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" event={"ID":"f00f570b-588f-4e4d-839f-423a7fa75cdf","Type":"ContainerStarted","Data":"374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78"}
Oct 30 19:25:52.117797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:52.117371    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" event={"ID":"f00f570b-588f-4e4d-839f-423a7fa75cdf","Type":"ContainerStarted","Data":"20f3d355f98f3cc1a9db5e16ed2f977a50ab218735cb5f42d514be6c59335c1f"}
Oct 30 19:25:52.128511 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:25:52.128481    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:25:52.144103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:52.144052    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" podStartSLOduration=2.144038105 podStartE2EDuration="2.144038105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:52.13897646 +0000 UTC m=+1187.041989487" watchObservedRunningTime="2026-10-30 19:25:52.144038105 +0000 UTC m=+1187.047051142"
Oct 30 19:25:52.377449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:52.377107    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:52.981146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:52.981110    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:53.071026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:53.070001    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:53.075543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:53.075521    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:53.125154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:53.125125    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:53.125958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:53.125938    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:25:54.026726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:54.017852    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:54.178228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:54.178187    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:54.345807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:54.345708    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:54.619802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:54.618847    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:54.816142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:54.816108    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:55.287016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:55.286981    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:55.742575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:55.742542    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:55.905751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:55.902959    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:56.139768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:56.138927    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:56.362215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:56.362184    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:56.956025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:56.955991    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:57.813284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:57.813251    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:58.439461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:58.436104    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:58.595531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:58.595492    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:59.082460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:59.082425    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:59.258360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:25:59.258323    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:00.068718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:00.065599    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:00.667327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:00.667296    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:01.031573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:01.031503    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"]
Oct 30 19:26:01.036091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:01.036070    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d620f5f-96f7b"]
Oct 30 19:26:01.268250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:01.268212    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc69199-346e-4c38-8c3a-2c7162b968e3" path="/var/lib/kubelet/pods/dfc69199-346e-4c38-8c3a-2c7162b968e3/volumes"
Oct 30 19:26:01.339037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:01.338964    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:01.510381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:01.510349    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:01.817570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:01.817536    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:02.058070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:02.058029    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"]
Oct 30 19:26:02.062613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:02.062583    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301bd19a-rdx5k"]
Oct 30 19:26:02.714891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:02.714848    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:03.175878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:03.175829    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:03.263441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:03.263410    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84e08a9-9727-4d86-9c25-d33155ca25d4" path="/var/lib/kubelet/pods/e84e08a9-9727-4d86-9c25-d33155ca25d4/volumes"
Oct 30 19:26:03.338067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:03.338025    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:04.037260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:04.037227    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:04.495984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:04.495939    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:04.665715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:04.665672    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:05.203414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:05.203376    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:05.386357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:05.386330    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:26:05.387857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:05.387839    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:26:06.038221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:06.038183    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:06.195484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:06.195401    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:06.549259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:06.549177    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:06.907575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:06.907442    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"]
Oct 30 19:26:06.914571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:06.914532    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:06.929902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:06.929876    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"]
Oct 30 19:26:06.931989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:06.931964    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-1e7123-openshift-default-dockercfg-m8qjk\""
Oct 30 19:26:07.013372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.013335    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-certs\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.014880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.014851    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-data\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.015176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.015152    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b50881-1ed8-48b6-9b45-44146b46a135-istiod-ca-cert\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.075780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.056865    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-istio-token\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.075780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.057137    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-envoy\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.075780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.057174    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wsvh\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-kube-api-access-7wsvh\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.075780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.057239    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-credential-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.075780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.057307    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.075780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.057367    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b50881-1ed8-48b6-9b45-44146b46a135-istio-podinfo\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.162768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.161160    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-envoy\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.162768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.161215    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wsvh\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-kube-api-access-7wsvh\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.162768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.161268    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-credential-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.162768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.161313    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.162768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.161359    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b50881-1ed8-48b6-9b45-44146b46a135-istio-podinfo\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.162768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.161474    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-certs\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.162768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.161554    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-data\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.162768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.161605    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b50881-1ed8-48b6-9b45-44146b46a135-istiod-ca-cert\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.162768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.161635    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-istio-token\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.170469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.170433    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-credential-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.171056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.171033    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.171545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.171526    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-data\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.187791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.173533    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b50881-1ed8-48b6-9b45-44146b46a135-istiod-ca-cert\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.188021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.173818    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-certs\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.188136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.185068    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-envoy\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.195307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.195240    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-istio-token\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.205408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.205346    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wsvh\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-kube-api-access-7wsvh\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.206149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.206127    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b50881-1ed8-48b6-9b45-44146b46a135-istio-podinfo\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.235960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.235659    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:07.308129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.304914    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:07.582571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.582471    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:07.613792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.613764    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"]
Oct 30 19:26:07.614261 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:26:07.614237    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b50881_1ed8_48b6_9b45_44146b46a135.slice/crio-1758e20c3a9dc347d59e5d1e85ae3d2b8e47c909a4e0e778839993d80bc7d27f WatchSource:0}: Error finding container 1758e20c3a9dc347d59e5d1e85ae3d2b8e47c909a4e0e778839993d80bc7d27f: Status 404 returned error can't find the container with id 1758e20c3a9dc347d59e5d1e85ae3d2b8e47c909a4e0e778839993d80bc7d27f
Oct 30 19:26:07.617052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.617019    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:07.617126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.617086    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:07.617126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:07.617114    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:08.338082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:08.337367    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:08.358068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:08.358032    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" event={"ID":"68b50881-1ed8-48b6-9b45-44146b46a135","Type":"ContainerStarted","Data":"cff9888bdeb8e4abb55c5f768029b401d41054d39d3749a72a5a7e05d6414137"}
Oct 30 19:26:08.358268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:08.358089    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" event={"ID":"68b50881-1ed8-48b6-9b45-44146b46a135","Type":"ContainerStarted","Data":"1758e20c3a9dc347d59e5d1e85ae3d2b8e47c909a4e0e778839993d80bc7d27f"}
Oct 30 19:26:08.383413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:08.383365    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" podStartSLOduration=2.383350097 podStartE2EDuration="2.383350097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:08.380113265 +0000 UTC m=+1203.283126294" watchObservedRunningTime="2026-10-30 19:26:08.383350097 +0000 UTC m=+1203.286363134"
Oct 30 19:26:09.209242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:09.209210    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:09.237114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:09.237055    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:09.242761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:09.242729    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:09.365980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:09.365941    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:09.367007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:09.366985    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:10.353680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:10.353649    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:10.788410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:10.788374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:10.959034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:10.958994    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:11.177630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:11.177592    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:11.975790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:11.975756    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:12.208394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:12.208254    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:13.443392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:13.443352    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:13.608119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:13.608050    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:13.789827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:13.789750    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:14.672756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:14.672720    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:14.874424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:14.874363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:15.030216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:15.030128    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"]
Oct 30 19:26:15.034280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:15.034254    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f54d2173-2lpgb"]
Oct 30 19:26:15.216224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:15.216197    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:15.263892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:15.263864    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5487022-9741-44d0-961b-3b60fe18fd78" path="/var/lib/kubelet/pods/c5487022-9741-44d0-961b-3b60fe18fd78/volumes"
Oct 30 19:26:15.580854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:15.580817    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:15.748894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:15.748863    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:16.459783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:16.459630    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:17.275366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:17.275336    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:17.942887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:17.942850    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:18.785714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:18.779306    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:19.287405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:19.287365    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:19.487271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:19.487243    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:19.829320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:19.829281    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:20.533052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.533021    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"]
Oct 30 19:26:20.533588 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.533560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:20.539363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.539345    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:20.544762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.544737    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e3586f-dockercfg-mkk5h\""
Oct 30 19:26:20.549805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.549778    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"]
Oct 30 19:26:20.582948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.582916    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e3586f\""
Oct 30 19:26:20.629601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.629545    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfqq8\" (UniqueName: \"kubernetes.io/projected/c91cc351-9794-443a-8a92-01b9239e06a4-kube-api-access-lfqq8\") pod \"payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7\" (UID: \"c91cc351-9794-443a-8a92-01b9239e06a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:20.629874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.629850    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91cc351-9794-443a-8a92-01b9239e06a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7\" (UID: \"c91cc351-9794-443a-8a92-01b9239e06a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:20.794773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.769332    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91cc351-9794-443a-8a92-01b9239e06a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7\" (UID: \"c91cc351-9794-443a-8a92-01b9239e06a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:20.910632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.910586    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfqq8\" (UniqueName: \"kubernetes.io/projected/c91cc351-9794-443a-8a92-01b9239e06a4-kube-api-access-lfqq8\") pod \"payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7\" (UID: \"c91cc351-9794-443a-8a92-01b9239e06a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:20.911160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.806460    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91cc351-9794-443a-8a92-01b9239e06a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7\" (UID: \"c91cc351-9794-443a-8a92-01b9239e06a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:20.911160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.804278    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:20.924927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:20.924899    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfqq8\" (UniqueName: \"kubernetes.io/projected/c91cc351-9794-443a-8a92-01b9239e06a4-kube-api-access-lfqq8\") pod \"payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7\" (UID: \"c91cc351-9794-443a-8a92-01b9239e06a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:21.124223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:21.124118    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:21.158394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:21.158363    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:21.478440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:21.475820    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"]
Oct 30 19:26:21.485898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:21.485870    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:21.498129 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:26:21.498080    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91cc351_9794_443a_8a92_01b9239e06a4.slice/crio-25123a449bc5a849d568bf5b953081a5f1087f87d01cef47145bf769e79dfd46 WatchSource:0}: Error finding container 25123a449bc5a849d568bf5b953081a5f1087f87d01cef47145bf769e79dfd46: Status 404 returned error can't find the container with id 25123a449bc5a849d568bf5b953081a5f1087f87d01cef47145bf769e79dfd46
Oct 30 19:26:22.127617 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:22.127585    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:26:22.247428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:22.247394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:22.487444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:22.487406    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7" event={"ID":"c91cc351-9794-443a-8a92-01b9239e06a4","Type":"ContainerStarted","Data":"a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6"}
Oct 30 19:26:22.487615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:22.487459    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7" event={"ID":"c91cc351-9794-443a-8a92-01b9239e06a4","Type":"ContainerStarted","Data":"25123a449bc5a849d568bf5b953081a5f1087f87d01cef47145bf769e79dfd46"}
Oct 30 19:26:22.487615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:22.487480    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:22.507518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:22.507469    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7" podStartSLOduration=2.507456201 podStartE2EDuration="2.507456201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:22.504734589 +0000 UTC m=+1217.407747625" watchObservedRunningTime="2026-10-30 19:26:22.507456201 +0000 UTC m=+1217.410469248"
Oct 30 19:26:23.052270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:23.052040    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:23.787197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:23.787069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:24.557964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:24.557935    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:25.505389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:25.505356    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:25.988787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:25.988746    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:26.187882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:26.187848    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:26.462042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:26.462004    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:27.097818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:27.097783    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:27.872911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:27.872874    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:28.398999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:28.398964    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:28.592201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:28.592162    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:29.323781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:29.323752    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:29.577220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:29.576615    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:30.097057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:30.097009    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:30.676175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:30.676144    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:31.505101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:31.505069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:32.216387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:32.216354    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:33.371944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:33.371894    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:33.497682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:33.497656    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:26:33.969991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:33.969960    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:34.849402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:34.849358    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:35.719981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:35.719943    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:36.233712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:36.227321    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:36.722361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:36.722324    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:36.915228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:36.915194    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:37.185313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:37.185272    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8861-747b36"]
Oct 30 19:26:37.190313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:37.190282    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:37.217120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:37.217090    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8861-747b36"]
Oct 30 19:26:37.242185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:37.232965    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bmmlc\""
Oct 30 19:26:37.356715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:37.328959    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92qcq\" (UniqueName: \"kubernetes.io/projected/1709362f-4ab6-49c5-b9ab-87c7313efe05-kube-api-access-92qcq\") pod \"test-curl-8861-747b36\" (UID: \"1709362f-4ab6-49c5-b9ab-87c7313efe05\") " pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:37.468756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:37.468643    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qcq\" (UniqueName: \"kubernetes.io/projected/1709362f-4ab6-49c5-b9ab-87c7313efe05-kube-api-access-92qcq\") pod \"test-curl-8861-747b36\" (UID: \"1709362f-4ab6-49c5-b9ab-87c7313efe05\") " pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:37.481467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:37.481439    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qcq\" (UniqueName: \"kubernetes.io/projected/1709362f-4ab6-49c5-b9ab-87c7313efe05-kube-api-access-92qcq\") pod \"test-curl-8861-747b36\" (UID: \"1709362f-4ab6-49c5-b9ab-87c7313efe05\") " pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:37.492563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:37.492535    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:37.556337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:37.556304    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:38.086460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:38.086415    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:38.158921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:38.158597    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8861-747b36"]
Oct 30 19:26:38.372435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:38.372345    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:38.576269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:38.576111    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:38.753368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:38.753336    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8861-747b36" event={"ID":"1709362f-4ab6-49c5-b9ab-87c7313efe05","Type":"ContainerStarted","Data":"e01e7f975776eeec3e84cedbff059df7c215fa3d96fe72babf6c6a379ceaf3a8"}
Oct 30 19:26:38.774300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:38.774272    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:39.177498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.177445    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:39.369307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.369275    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:39.852713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.838838    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"]
Oct 30 19:26:39.852713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.844284    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:39.854277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.854131    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:39.854277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.854246    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:26:39.869575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.868794    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"]
Oct 30 19:26:39.961272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.960757    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bd658e8-x5l8x\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:39.961272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.960978    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxqx\" (UniqueName: \"kubernetes.io/projected/719a9674-bf9f-4f41-89d2-697cedb57e0e-kube-api-access-mmxqx\") pod \"maas-api-revoke-sub-8bd658e8-x5l8x\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:39.961272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:39.961030    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bd658e8-x5l8x\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:40.078158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.076083    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxqx\" (UniqueName: \"kubernetes.io/projected/719a9674-bf9f-4f41-89d2-697cedb57e0e-kube-api-access-mmxqx\") pod \"maas-api-revoke-sub-8bd658e8-x5l8x\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:40.078158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.076162    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bd658e8-x5l8x\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:40.078158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.076396    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bd658e8-x5l8x\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:40.078158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.077551    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bd658e8-x5l8x\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:40.081946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.081921    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bd658e8-x5l8x\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:40.090539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.090513    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxqx\" (UniqueName: \"kubernetes.io/projected/719a9674-bf9f-4f41-89d2-697cedb57e0e-kube-api-access-mmxqx\") pod \"maas-api-revoke-sub-8bd658e8-x5l8x\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:40.186339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.186305    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:40.315101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.314949    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:40.794124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.794039    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:40.893433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:40.893403    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"]
Oct 30 19:26:41.018006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:41.016296    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:41.242995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:41.242965    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:41.895643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:41.888380    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x" event={"ID":"719a9674-bf9f-4f41-89d2-697cedb57e0e","Type":"ContainerStarted","Data":"cd9827738de6298261f3e3fa57160bb8eb5419988b098f68f4f5e04e53085d20"}
Oct 30 19:26:41.895643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:41.888469    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x" event={"ID":"719a9674-bf9f-4f41-89d2-697cedb57e0e","Type":"ContainerStarted","Data":"d75f0a01e381de62101d4cffe394c1e18ca89cb74e0a928bd244b6d7c271b8c9"}
Oct 30 19:26:41.950742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:41.934626    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x" podStartSLOduration=2.934603954 podStartE2EDuration="2.934603954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:41.924089671 +0000 UTC m=+1236.827102709" watchObservedRunningTime="2026-10-30 19:26:41.934603954 +0000 UTC m=+1236.837616992"
Oct 30 19:26:42.035220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:42.035130    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:42.236537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:42.236501    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:42.930009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:42.929974    4925 generic.go:356] "Generic (PLEG): container finished" podID="719a9674-bf9f-4f41-89d2-697cedb57e0e" containerID="cd9827738de6298261f3e3fa57160bb8eb5419988b098f68f4f5e04e53085d20" exitCode=0
Oct 30 19:26:42.930495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:42.930159    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x" event={"ID":"719a9674-bf9f-4f41-89d2-697cedb57e0e","Type":"ContainerDied","Data":"cd9827738de6298261f3e3fa57160bb8eb5419988b098f68f4f5e04e53085d20"}
Oct 30 19:26:43.054021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:43.053985    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:43.261552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:43.261460    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:44.049871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.049480    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:44.171124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.171099    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:44.287385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.286309    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719a9674-bf9f-4f41-89d2-697cedb57e0e-kube-api-access-mmxqx\" (UniqueName: \"kubernetes.io/projected/719a9674-bf9f-4f41-89d2-697cedb57e0e-kube-api-access-mmxqx\") pod \"719a9674-bf9f-4f41-89d2-697cedb57e0e\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") "
Oct 30 19:26:44.287385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.286455    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-xks-ca\") pod \"719a9674-bf9f-4f41-89d2-697cedb57e0e\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") "
Oct 30 19:26:44.287385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.286486    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-openshift-ca\") pod \"719a9674-bf9f-4f41-89d2-697cedb57e0e\" (UID: \"719a9674-bf9f-4f41-89d2-697cedb57e0e\") "
Oct 30 19:26:44.288189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.287793    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-openshift-ca" pod "719a9674-bf9f-4f41-89d2-697cedb57e0e" (UID: "719a9674-bf9f-4f41-89d2-697cedb57e0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:44.295197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.295166    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719a9674-bf9f-4f41-89d2-697cedb57e0e-kube-api-access-mmxqx" pod "719a9674-bf9f-4f41-89d2-697cedb57e0e" (UID: "719a9674-bf9f-4f41-89d2-697cedb57e0e"). InnerVolumeSpecName "kube-api-access-mmxqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:44.297427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.297400    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-xks-ca" pod "719a9674-bf9f-4f41-89d2-697cedb57e0e" (UID: "719a9674-bf9f-4f41-89d2-697cedb57e0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:44.388853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.388819    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxqx\" (UniqueName: \"kubernetes.io/projected/719a9674-bf9f-4f41-89d2-697cedb57e0e-kube-api-access-mmxqx\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:44.388853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.388852    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:44.388853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.388865    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719a9674-bf9f-4f41-89d2-697cedb57e0e-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:44.417924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.417891    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"]
Oct 30 19:26:44.418908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.418888    4925 cpu_manager.go:414] "containerMap: removing container" podUID="719a9674-bf9f-4f41-89d2-697cedb57e0e" containerName="revoke-keys"
Oct 30 19:26:44.418972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.418911    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719a9674-bf9f-4f41-89d2-697cedb57e0e" containerName="revoke-keys"
Oct 30 19:26:44.419103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.419091    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="719a9674-bf9f-4f41-89d2-697cedb57e0e" containerName="revoke-keys"
Oct 30 19:26:44.423103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.423084    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:44.442002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.441969    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"]
Oct 30 19:26:44.442243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.442220    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-48b1a6a7\""
Oct 30 19:26:44.443966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.443937    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:44.488734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.487862    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"]
Oct 30 19:26:44.492542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.490452    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scw8h\" (UniqueName: \"kubernetes.io/projected/25348f9a-6e93-4fb7-b127-51b6910e514b-kube-api-access-scw8h\") pod \"payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz\" (UID: \"25348f9a-6e93-4fb7-b127-51b6910e514b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:44.492542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.490578    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25348f9a-6e93-4fb7-b127-51b6910e514b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz\" (UID: \"25348f9a-6e93-4fb7-b127-51b6910e514b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:44.493404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.493388    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:44.506813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.506777    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-48b1a6a7-dockercfg-2zj4p\""
Oct 30 19:26:44.508181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.508159    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"]
Oct 30 19:26:44.592264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.592177    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scw8h\" (UniqueName: \"kubernetes.io/projected/25348f9a-6e93-4fb7-b127-51b6910e514b-kube-api-access-scw8h\") pod \"payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz\" (UID: \"25348f9a-6e93-4fb7-b127-51b6910e514b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:44.592439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.592306    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25348f9a-6e93-4fb7-b127-51b6910e514b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz\" (UID: \"25348f9a-6e93-4fb7-b127-51b6910e514b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:44.592439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.592372    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnbmr\" (UniqueName: \"kubernetes.io/projected/dd4240a3-839f-479d-83c9-526adaef0013-kube-api-access-hnbmr\") pod \"payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs\" (UID: \"dd4240a3-839f-479d-83c9-526adaef0013\") " pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:44.592559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.592532    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd4240a3-839f-479d-83c9-526adaef0013-plugins-config-volume\") pod \"payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs\" (UID: \"dd4240a3-839f-479d-83c9-526adaef0013\") " pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:44.593341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.593316    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25348f9a-6e93-4fb7-b127-51b6910e514b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz\" (UID: \"25348f9a-6e93-4fb7-b127-51b6910e514b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:44.608661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.608635    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scw8h\" (UniqueName: \"kubernetes.io/projected/25348f9a-6e93-4fb7-b127-51b6910e514b-kube-api-access-scw8h\") pod \"payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz\" (UID: \"25348f9a-6e93-4fb7-b127-51b6910e514b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:44.620078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.620054    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:44.704711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.697824    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnbmr\" (UniqueName: \"kubernetes.io/projected/dd4240a3-839f-479d-83c9-526adaef0013-kube-api-access-hnbmr\") pod \"payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs\" (UID: \"dd4240a3-839f-479d-83c9-526adaef0013\") " pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:44.704711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.698833    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd4240a3-839f-479d-83c9-526adaef0013-plugins-config-volume\") pod \"payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs\" (UID: \"dd4240a3-839f-479d-83c9-526adaef0013\") " pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:44.704711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.699162    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd4240a3-839f-479d-83c9-526adaef0013-plugins-config-volume\") pod \"payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs\" (UID: \"dd4240a3-839f-479d-83c9-526adaef0013\") " pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:44.742710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.739673    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnbmr\" (UniqueName: \"kubernetes.io/projected/dd4240a3-839f-479d-83c9-526adaef0013-kube-api-access-hnbmr\") pod \"payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs\" (UID: \"dd4240a3-839f-479d-83c9-526adaef0013\") " pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:44.760712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.752950    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:44.799713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.789199    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:44.852813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:44.831279    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:45.008705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.003707    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8861-747b36" event={"ID":"1709362f-4ab6-49c5-b9ab-87c7313efe05","Type":"ContainerStarted","Data":"e969f929ce68a46c56a05978a6a43fd5e40da92eeb0d6a1ebf5222a098bab82f"}
Oct 30 19:26:45.008705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.006634    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"
Oct 30 19:26:45.008705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.006779    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x" event={"ID":"719a9674-bf9f-4f41-89d2-697cedb57e0e","Type":"ContainerDied","Data":"d75f0a01e381de62101d4cffe394c1e18ca89cb74e0a928bd244b6d7c271b8c9"}
Oct 30 19:26:45.008705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.006832    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d75f0a01e381de62101d4cffe394c1e18ca89cb74e0a928bd244b6d7c271b8c9"
Oct 30 19:26:45.024001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.023941    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8861-747b36" podStartSLOduration=2.133760557 podStartE2EDuration="8.023922116s" totalImagesPullingTime="5.890161559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:45.021263786 +0000 UTC m=+1239.924276825" watchObservedRunningTime="2026-10-30 19:26:45.023922116 +0000 UTC m=+1239.926935154"
Oct 30 19:26:45.179464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.179424    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"]
Oct 30 19:26:45.214978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.214951    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"]
Oct 30 19:26:45.217236 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:26:45.217209    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4240a3_839f_479d_83c9_526adaef0013.slice/crio-7c4be612c0bcec38cd4a3da1dcaf88a341174a70ba518c83ee087ed4c554ef92 WatchSource:0}: Error finding container 7c4be612c0bcec38cd4a3da1dcaf88a341174a70ba518c83ee087ed4c554ef92: Status 404 returned error can't find the container with id 7c4be612c0bcec38cd4a3da1dcaf88a341174a70ba518c83ee087ed4c554ef92
Oct 30 19:26:45.298380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.298346    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:45.853741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.853699    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8861-747b36"]
Oct 30 19:26:45.860778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:45.860300    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8861-747b36"]
Oct 30 19:26:46.019851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.019816    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs" event={"ID":"dd4240a3-839f-479d-83c9-526adaef0013","Type":"ContainerStarted","Data":"0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde"}
Oct 30 19:26:46.020027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.019921    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs" event={"ID":"dd4240a3-839f-479d-83c9-526adaef0013","Type":"ContainerStarted","Data":"7c4be612c0bcec38cd4a3da1dcaf88a341174a70ba518c83ee087ed4c554ef92"}
Oct 30 19:26:46.020027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.019953    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:46.022182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.022139    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz" event={"ID":"25348f9a-6e93-4fb7-b127-51b6910e514b","Type":"ContainerStarted","Data":"2aaf9ff8118e511e11efa78c953050da72ceb4eb489e5d3fc5bb4d8ab15fd8de"}
Oct 30 19:26:46.022345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.022197    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz" event={"ID":"25348f9a-6e93-4fb7-b127-51b6910e514b","Type":"ContainerStarted","Data":"0ca1dd5c13fcd386e365f26fe323d195ffd03581e9861aa78b43c10c4b7dfb2c"}
Oct 30 19:26:46.022345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.022264    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:46.030066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.030044    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8861-9b558c"]
Oct 30 19:26:46.030680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.030659    4925 cpu_manager.go:414] "containerMap: removing container" podUID="1709362f-4ab6-49c5-b9ab-87c7313efe05" containerName="test-curl-8861-747b36"
Oct 30 19:26:46.030680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.030679    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1709362f-4ab6-49c5-b9ab-87c7313efe05" containerName="test-curl-8861-747b36"
Oct 30 19:26:46.030931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.030916    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="1709362f-4ab6-49c5-b9ab-87c7313efe05" containerName="test-curl-8861-747b36"
Oct 30 19:26:46.036222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.036200    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-9b558c"
Oct 30 19:26:46.085059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.047825    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs" podStartSLOduration=2.047803888 podStartE2EDuration="2.047803888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:46.038714597 +0000 UTC m=+1240.941727637" watchObservedRunningTime="2026-10-30 19:26:46.047803888 +0000 UTC m=+1240.950816925"
Oct 30 19:26:46.085059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.063511    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8861-9b558c"]
Oct 30 19:26:46.099422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.099389    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:46.102825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.102766    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz" podStartSLOduration=2.102746948 podStartE2EDuration="2.102746948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:46.063248813 +0000 UTC m=+1240.966261850" watchObservedRunningTime="2026-10-30 19:26:46.102746948 +0000 UTC m=+1241.005759989"
Oct 30 19:26:46.147516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.147479    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6669d\" (UniqueName: \"kubernetes.io/projected/38594aa4-faee-485e-8035-e9b774fe766e-kube-api-access-6669d\") pod \"test-curl-8861-9b558c\" (UID: \"38594aa4-faee-485e-8035-e9b774fe766e\") " pod="opendatahub/test-curl-8861-9b558c"
Oct 30 19:26:46.250043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.249605    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6669d\" (UniqueName: \"kubernetes.io/projected/38594aa4-faee-485e-8035-e9b774fe766e-kube-api-access-6669d\") pod \"test-curl-8861-9b558c\" (UID: \"38594aa4-faee-485e-8035-e9b774fe766e\") " pod="opendatahub/test-curl-8861-9b558c"
Oct 30 19:26:46.266761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.265177    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6669d\" (UniqueName: \"kubernetes.io/projected/38594aa4-faee-485e-8035-e9b774fe766e-kube-api-access-6669d\") pod \"test-curl-8861-9b558c\" (UID: \"38594aa4-faee-485e-8035-e9b774fe766e\") " pod="opendatahub/test-curl-8861-9b558c"
Oct 30 19:26:46.334085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.334052    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:46.408289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.408214    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-9b558c"
Oct 30 19:26:46.660994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.660907    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:46.741939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.741911    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8861-9b558c"]
Oct 30 19:26:46.743519 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:26:46.743496    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38594aa4_faee_485e_8035_e9b774fe766e.slice/crio-6f2f8e7294ff3bc29928f12e5e4e58899b8135de0bd0904ac1909935c87a8b65 WatchSource:0}: Error finding container 6f2f8e7294ff3bc29928f12e5e4e58899b8135de0bd0904ac1909935c87a8b65: Status 404 returned error can't find the container with id 6f2f8e7294ff3bc29928f12e5e4e58899b8135de0bd0904ac1909935c87a8b65
Oct 30 19:26:46.831601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:46.831576    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:47.037459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:47.037366    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8861-9b558c" event={"ID":"38594aa4-faee-485e-8035-e9b774fe766e","Type":"ContainerStarted","Data":"6e1951d26aa722ad8ccc48cf0621a70fc888d4f7e3e1b68a20c2e6c70ed543de"}
Oct 30 19:26:47.037459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:47.037418    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8861-9b558c" event={"ID":"38594aa4-faee-485e-8035-e9b774fe766e","Type":"ContainerStarted","Data":"6f2f8e7294ff3bc29928f12e5e4e58899b8135de0bd0904ac1909935c87a8b65"}
Oct 30 19:26:47.037743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:47.037546    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8861-747b36" podUID="1709362f-4ab6-49c5-b9ab-87c7313efe05" containerName="test-curl-8861-747b36" containerID="cri-o://e969f929ce68a46c56a05978a6a43fd5e40da92eeb0d6a1ebf5222a098bab82f" gracePeriod=2
Oct 30 19:26:47.041409 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:47.041380    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-747b36\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="1709362f-4ab6-49c5-b9ab-87c7313efe05" pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:47.042823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:47.042798    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:47.068160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:47.068113    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8861-9b558c" podStartSLOduration=1.068101158 podStartE2EDuration="1.068101158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:47.067958492 +0000 UTC m=+1241.970971529" watchObservedRunningTime="2026-10-30 19:26:47.068101158 +0000 UTC m=+1241.971114194"
Oct 30 19:26:47.070836 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:47.070808    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-747b36\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="1709362f-4ab6-49c5-b9ab-87c7313efe05" pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:47.932141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:47.932114    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:48.132942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.132849    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8861-9b558c"]
Oct 30 19:26:48.156969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.143097    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8861-9b558c"]
Oct 30 19:26:48.222649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.222558    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8861-795efb"]
Oct 30 19:26:48.226734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.225195    4925 cpu_manager.go:414] "containerMap: removing container" podUID="38594aa4-faee-485e-8035-e9b774fe766e" containerName="test-curl-8861-9b558c"
Oct 30 19:26:48.226734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.225220    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38594aa4-faee-485e-8035-e9b774fe766e" containerName="test-curl-8861-9b558c"
Oct 30 19:26:48.226734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.225442    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="38594aa4-faee-485e-8035-e9b774fe766e" containerName="test-curl-8861-9b558c"
Oct 30 19:26:48.228043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.227979    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:48.230761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.230061    4925 generic.go:356] "Generic (PLEG): container finished" podID="68b50881-1ed8-48b6-9b45-44146b46a135" containerID="cff9888bdeb8e4abb55c5f768029b401d41054d39d3749a72a5a7e05d6414137" exitCode=0
Oct 30 19:26:48.230761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.230153    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8861-795efb"]
Oct 30 19:26:48.230761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.230177    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" event={"ID":"68b50881-1ed8-48b6-9b45-44146b46a135","Type":"ContainerDied","Data":"cff9888bdeb8e4abb55c5f768029b401d41054d39d3749a72a5a7e05d6414137"}
Oct 30 19:26:48.230761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.230604    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-795efb"
Oct 30 19:26:48.230761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.230654    4925 scope.go:122] "RemoveContainer" containerID="cff9888bdeb8e4abb55c5f768029b401d41054d39d3749a72a5a7e05d6414137"
Oct 30 19:26:48.261965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.261884    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:48.262079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.262033    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:48.263373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.262141    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:48.320564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.320356    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjfl\" (UniqueName: \"kubernetes.io/projected/0879ff3c-0e8f-4682-9a68-47d88140a658-kube-api-access-rvjfl\") pod \"test-curl-8861-795efb\" (UID: \"0879ff3c-0e8f-4682-9a68-47d88140a658\") " pod="opendatahub/test-curl-8861-795efb"
Oct 30 19:26:48.422054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.422023    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjfl\" (UniqueName: \"kubernetes.io/projected/0879ff3c-0e8f-4682-9a68-47d88140a658-kube-api-access-rvjfl\") pod \"test-curl-8861-795efb\" (UID: \"0879ff3c-0e8f-4682-9a68-47d88140a658\") " pod="opendatahub/test-curl-8861-795efb"
Oct 30 19:26:48.439682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.439652    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjfl\" (UniqueName: \"kubernetes.io/projected/0879ff3c-0e8f-4682-9a68-47d88140a658-kube-api-access-rvjfl\") pod \"test-curl-8861-795efb\" (UID: \"0879ff3c-0e8f-4682-9a68-47d88140a658\") " pod="opendatahub/test-curl-8861-795efb"
Oct 30 19:26:48.578758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.578722    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-795efb"
Oct 30 19:26:48.868172 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:26:48.868143    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0879ff3c_0e8f_4682_9a68_47d88140a658.slice/crio-734ed3dcf8385c87cbfd97bbd361b0c8f0ce945426d1260e46b91de13d35da27 WatchSource:0}: Error finding container 734ed3dcf8385c87cbfd97bbd361b0c8f0ce945426d1260e46b91de13d35da27: Status 404 returned error can't find the container with id 734ed3dcf8385c87cbfd97bbd361b0c8f0ce945426d1260e46b91de13d35da27
Oct 30 19:26:48.868745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:48.868721    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8861-795efb"]
Oct 30 19:26:49.168289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.168253    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:49.281048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.281024    4925 generic.go:356] "Generic (PLEG): container finished" podID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerID="374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78" exitCode=0
Oct 30 19:26:49.281485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.281463    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" event={"ID":"68b50881-1ed8-48b6-9b45-44146b46a135","Type":"ContainerStarted","Data":"84ac7c5cd4020f1ac6cc7949cda813948e5b9bda3c221a88839026740b721307"}
Oct 30 19:26:49.281584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.281510    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8861-795efb" event={"ID":"0879ff3c-0e8f-4682-9a68-47d88140a658","Type":"ContainerStarted","Data":"43a80654cff3e7e982e42335168fdecb9c6f7286610e9870821169ce28a01d9c"}
Oct 30 19:26:49.281584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.281530    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8861-795efb" event={"ID":"0879ff3c-0e8f-4682-9a68-47d88140a658","Type":"ContainerStarted","Data":"734ed3dcf8385c87cbfd97bbd361b0c8f0ce945426d1260e46b91de13d35da27"}
Oct 30 19:26:49.281584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.281541    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" event={"ID":"f00f570b-588f-4e4d-839f-423a7fa75cdf","Type":"ContainerDied","Data":"374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78"}
Oct 30 19:26:49.281985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.281967    4925 scope.go:122] "RemoveContainer" containerID="374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78"
Oct 30 19:26:49.284070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.282671    4925 generic.go:356] "Generic (PLEG): container finished" podID="1709362f-4ab6-49c5-b9ab-87c7313efe05" containerID="e969f929ce68a46c56a05978a6a43fd5e40da92eeb0d6a1ebf5222a098bab82f" exitCode=137
Oct 30 19:26:49.284070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.282929    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8861-9b558c" podUID="38594aa4-faee-485e-8035-e9b774fe766e" containerName="test-curl-8861-9b558c" containerID="cri-o://6e1951d26aa722ad8ccc48cf0621a70fc888d4f7e3e1b68a20c2e6c70ed543de" gracePeriod=2
Oct 30 19:26:49.287043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.286800    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:49.287043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.286884    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:49.287043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.286931    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:49.326815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.316256    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8861-795efb" podStartSLOduration=1.316231894 podStartE2EDuration="1.316231894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:49.2995369 +0000 UTC m=+1244.202549951" watchObservedRunningTime="2026-10-30 19:26:49.316231894 +0000 UTC m=+1244.219244932"
Oct 30 19:26:49.335955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.335929    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:49.439979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.438127    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1709362f-4ab6-49c5-b9ab-87c7313efe05-kube-api-access-92qcq\" (UniqueName: \"kubernetes.io/projected/1709362f-4ab6-49c5-b9ab-87c7313efe05-kube-api-access-92qcq\") pod \"1709362f-4ab6-49c5-b9ab-87c7313efe05\" (UID: \"1709362f-4ab6-49c5-b9ab-87c7313efe05\") "
Oct 30 19:26:49.441992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.441967    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1709362f-4ab6-49c5-b9ab-87c7313efe05-kube-api-access-92qcq" pod "1709362f-4ab6-49c5-b9ab-87c7313efe05" (UID: "1709362f-4ab6-49c5-b9ab-87c7313efe05"). InnerVolumeSpecName "kube-api-access-92qcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:49.540550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.540516    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92qcq\" (UniqueName: \"kubernetes.io/projected/1709362f-4ab6-49c5-b9ab-87c7313efe05-kube-api-access-92qcq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:49.729730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.729614    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"]
Oct 30 19:26:49.734828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.734799    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:49.737597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.737572    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"]
Oct 30 19:26:49.739891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.739866    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:49.739891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.739881    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:26:49.851750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.846926    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bb54977-4vmp9\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:49.851750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.847014    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrc8\" (UniqueName: \"kubernetes.io/projected/cc4a8d39-c822-4e84-8c2d-7f57bc070439-kube-api-access-lhrc8\") pod \"maas-api-revoke-sub-6bb54977-4vmp9\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:49.851750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.847072    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bb54977-4vmp9\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:49.874512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.871195    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:49.948826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.948792    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bb54977-4vmp9\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:49.949011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.948870    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrc8\" (UniqueName: \"kubernetes.io/projected/cc4a8d39-c822-4e84-8c2d-7f57bc070439-kube-api-access-lhrc8\") pod \"maas-api-revoke-sub-6bb54977-4vmp9\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:49.949011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.948917    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bb54977-4vmp9\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:49.949522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.949498    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bb54977-4vmp9\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:49.951674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.951647    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bb54977-4vmp9\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:49.961451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:49.961411    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrc8\" (UniqueName: \"kubernetes.io/projected/cc4a8d39-c822-4e84-8c2d-7f57bc070439-kube-api-access-lhrc8\") pod \"maas-api-revoke-sub-6bb54977-4vmp9\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:50.060613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.060133    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"]
Oct 30 19:26:50.063682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.063656    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10779f52-qvc2r"]
Oct 30 19:26:50.086433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.086411    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:50.141010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.140976    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8861-795efb"]
Oct 30 19:26:50.163762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.163735    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8861-795efb"]
Oct 30 19:26:50.252327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.251833    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:50.252327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.251984    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:50.252327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.252028    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:26:50.255465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.255439    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:50.280148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.278616    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8861-ca1e7f"]
Oct 30 19:26:50.280148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.279750    4925 cpu_manager.go:414] "containerMap: removing container" podUID="0879ff3c-0e8f-4682-9a68-47d88140a658" containerName="test-curl-8861-795efb"
Oct 30 19:26:50.280148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.279772    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0879ff3c-0e8f-4682-9a68-47d88140a658" containerName="test-curl-8861-795efb"
Oct 30 19:26:50.280148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.280008    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0879ff3c-0e8f-4682-9a68-47d88140a658" containerName="test-curl-8861-795efb"
Oct 30 19:26:50.293304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.293270    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:50.352853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.349456    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8861-ca1e7f"]
Oct 30 19:26:50.366104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.366072    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" event={"ID":"f00f570b-588f-4e4d-839f-423a7fa75cdf","Type":"ContainerStarted","Data":"36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c"}
Oct 30 19:26:50.368169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.368150    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:50.368294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.368175    4925 scope.go:122] "RemoveContainer" containerID="e969f929ce68a46c56a05978a6a43fd5e40da92eeb0d6a1ebf5222a098bab82f"
Oct 30 19:26:50.393636 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:50.393600    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-747b36\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="1709362f-4ab6-49c5-b9ab-87c7313efe05" pod="opendatahub/test-curl-8861-747b36"
Oct 30 19:26:50.465274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.464257    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wzh\" (UniqueName: \"kubernetes.io/projected/26193413-d9e4-4ec5-b532-5c3e3583f3e4-kube-api-access-99wzh\") pod \"test-curl-8861-ca1e7f\" (UID: \"26193413-d9e4-4ec5-b532-5c3e3583f3e4\") " pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:50.566747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.566038    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wzh\" (UniqueName: \"kubernetes.io/projected/26193413-d9e4-4ec5-b532-5c3e3583f3e4-kube-api-access-99wzh\") pod \"test-curl-8861-ca1e7f\" (UID: \"26193413-d9e4-4ec5-b532-5c3e3583f3e4\") " pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:50.579999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.579249    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:50.584756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.584722    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"]
Oct 30 19:26:50.587628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.587509    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wzh\" (UniqueName: \"kubernetes.io/projected/26193413-d9e4-4ec5-b532-5c3e3583f3e4-kube-api-access-99wzh\") pod \"test-curl-8861-ca1e7f\" (UID: \"26193413-d9e4-4ec5-b532-5c3e3583f3e4\") " pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:50.672728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:50.672700    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:51.054731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.052981    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8861-ca1e7f"]
Oct 30 19:26:51.055676 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:26:51.055645    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26193413_d9e4_4ec5_b532_5c3e3583f3e4.slice/crio-3879132b4c681fdcbbcc9301a0d06b44a56ea1fa4e3ff719db4d977213a7a0e4 WatchSource:0}: Error finding container 3879132b4c681fdcbbcc9301a0d06b44a56ea1fa4e3ff719db4d977213a7a0e4: Status 404 returned error can't find the container with id 3879132b4c681fdcbbcc9301a0d06b44a56ea1fa4e3ff719db4d977213a7a0e4
Oct 30 19:26:51.069462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.069439    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:26:51.070427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.070401    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:26:51.070544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.070453    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:26:51.184668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.183962    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:51.253227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.239616    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:51.253227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.239663    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:26:51.271465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.270648    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1709362f-4ab6-49c5-b9ab-87c7313efe05" path="/var/lib/kubelet/pods/1709362f-4ab6-49c5-b9ab-87c7313efe05/volumes"
Oct 30 19:26:51.271465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.271180    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2303cb33-dadf-4461-afbe-c3d1ede0bde6" path="/var/lib/kubelet/pods/2303cb33-dadf-4461-afbe-c3d1ede0bde6/volumes"
Oct 30 19:26:51.380436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.380398    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9" event={"ID":"cc4a8d39-c822-4e84-8c2d-7f57bc070439","Type":"ContainerStarted","Data":"2bda1b613e580a21626c6db5fe31afa8add0fc888502d2db6e262bc49a7a6664"}
Oct 30 19:26:51.380576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.380450    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9" event={"ID":"cc4a8d39-c822-4e84-8c2d-7f57bc070439","Type":"ContainerStarted","Data":"fa94d1cf390b44cf3acab59b9e212cfc2200e464cce65669bc6cbbb7aff2c531"}
Oct 30 19:26:51.382794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.382772    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8861-ca1e7f" event={"ID":"26193413-d9e4-4ec5-b532-5c3e3583f3e4","Type":"ContainerStarted","Data":"2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6"}
Oct 30 19:26:51.382866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.382804    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8861-ca1e7f" event={"ID":"26193413-d9e4-4ec5-b532-5c3e3583f3e4","Type":"ContainerStarted","Data":"3879132b4c681fdcbbcc9301a0d06b44a56ea1fa4e3ff719db4d977213a7a0e4"}
Oct 30 19:26:51.385306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.385284    4925 generic.go:356] "Generic (PLEG): container finished" podID="38594aa4-faee-485e-8035-e9b774fe766e" containerID="6e1951d26aa722ad8ccc48cf0621a70fc888d4f7e3e1b68a20c2e6c70ed543de" exitCode=137
Oct 30 19:26:51.386008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.385980    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8861-795efb" podUID="0879ff3c-0e8f-4682-9a68-47d88140a658" containerName="test-curl-8861-795efb" containerID="cri-o://43a80654cff3e7e982e42335168fdecb9c6f7286610e9870821169ce28a01d9c" gracePeriod=2
Oct 30 19:26:51.459431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.459368    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8861-ca1e7f" podStartSLOduration=1.459347626 podStartE2EDuration="1.459347626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:51.457327177 +0000 UTC m=+1246.360340214" watchObservedRunningTime="2026-10-30 19:26:51.459347626 +0000 UTC m=+1246.362360663"
Oct 30 19:26:51.465310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.465259    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9" podStartSLOduration=2.465242404 podStartE2EDuration="2.465242404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:51.434738183 +0000 UTC m=+1246.337751274" watchObservedRunningTime="2026-10-30 19:26:51.465242404 +0000 UTC m=+1246.368255442"
Oct 30 19:26:51.657989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.657964    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-9b558c"
Oct 30 19:26:51.690501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.690476    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38594aa4-faee-485e-8035-e9b774fe766e-kube-api-access-6669d\" (UniqueName: \"kubernetes.io/projected/38594aa4-faee-485e-8035-e9b774fe766e-kube-api-access-6669d\") pod \"38594aa4-faee-485e-8035-e9b774fe766e\" (UID: \"38594aa4-faee-485e-8035-e9b774fe766e\") "
Oct 30 19:26:51.692866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.692846    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38594aa4-faee-485e-8035-e9b774fe766e-kube-api-access-6669d" pod "38594aa4-faee-485e-8035-e9b774fe766e" (UID: "38594aa4-faee-485e-8035-e9b774fe766e"). InnerVolumeSpecName "kube-api-access-6669d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:51.792454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.792417    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6669d\" (UniqueName: \"kubernetes.io/projected/38594aa4-faee-485e-8035-e9b774fe766e-kube-api-access-6669d\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:51.958655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:51.958593    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:52.073417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.072988    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:26:52.073417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.073063    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:26:52.178640 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:52.178609    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:26:52.244793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.244133    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:52.244793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.244194    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:26:52.353489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.353424    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8861-ca1e7f"]
Oct 30 19:26:52.357910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.357886    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8861-ca1e7f"]
Oct 30 19:26:52.452764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.451599    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-9b558c"
Oct 30 19:26:52.452764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.451737    4925 scope.go:122] "RemoveContainer" containerID="6e1951d26aa722ad8ccc48cf0621a70fc888d4f7e3e1b68a20c2e6c70ed543de"
Oct 30 19:26:52.458853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.458071    4925 generic.go:356] "Generic (PLEG): container finished" podID="cc4a8d39-c822-4e84-8c2d-7f57bc070439" containerID="2bda1b613e580a21626c6db5fe31afa8add0fc888502d2db6e262bc49a7a6664" exitCode=0
Oct 30 19:26:52.458853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.458589    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9" event={"ID":"cc4a8d39-c822-4e84-8c2d-7f57bc070439","Type":"ContainerDied","Data":"2bda1b613e580a21626c6db5fe31afa8add0fc888502d2db6e262bc49a7a6664"}
Oct 30 19:26:52.485586 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:52.485558    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-9b558c\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="38594aa4-faee-485e-8035-e9b774fe766e" pod="opendatahub/test-curl-8861-9b558c"
Oct 30 19:26:52.846791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:52.841731    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:53.069471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.069442    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:26:53.069652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.069496    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:26:53.126764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.125801    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:26:53.186396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.186354    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:53.249712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.237053    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:53.249712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.237114    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:26:53.265645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.265614    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38594aa4-faee-485e-8035-e9b774fe766e" path="/var/lib/kubelet/pods/38594aa4-faee-485e-8035-e9b774fe766e/volumes"
Oct 30 19:26:53.470034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.469998    4925 generic.go:356] "Generic (PLEG): container finished" podID="0879ff3c-0e8f-4682-9a68-47d88140a658" containerID="43a80654cff3e7e982e42335168fdecb9c6f7286610e9870821169ce28a01d9c" exitCode=137
Oct 30 19:26:53.470420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.470206    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8861-ca1e7f" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" containerName="test-curl-8861-ca1e7f" containerID="cri-o://2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6" gracePeriod=2
Oct 30 19:26:53.964059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:53.921087    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:54.115766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.071288    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:26:54.115766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.071343    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:26:54.204495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.204444    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:54.242766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.241070    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:54.242766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.241126    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:26:54.378713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.370532    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:54.775151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.774967    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:54.860980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.859657    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:26:54.952742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.902904    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:54.988765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.969937    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:54.994834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:54.994798    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:26:55.039813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.028501    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:55.039813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.029659    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:26:55.043075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.043042    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:55.057167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.056896    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-795efb"
Oct 30 19:26:55.075135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.075097    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:26:55.075397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.075369    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:26:55.115261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.101072    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0879ff3c-0e8f-4682-9a68-47d88140a658-kube-api-access-rvjfl\" (UniqueName: \"kubernetes.io/projected/0879ff3c-0e8f-4682-9a68-47d88140a658-kube-api-access-rvjfl\") pod \"0879ff3c-0e8f-4682-9a68-47d88140a658\" (UID: \"0879ff3c-0e8f-4682-9a68-47d88140a658\") "
Oct 30 19:26:55.115261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.105227    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0879ff3c-0e8f-4682-9a68-47d88140a658-kube-api-access-rvjfl" pod "0879ff3c-0e8f-4682-9a68-47d88140a658" (UID: "0879ff3c-0e8f-4682-9a68-47d88140a658"). InnerVolumeSpecName "kube-api-access-rvjfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:55.143111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.142890    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:55.169612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.169263    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:55.207782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.207258    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-xks-ca\") pod \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") "
Oct 30 19:26:55.208150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.208131    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-openshift-ca\") pod \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") "
Oct 30 19:26:55.208429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.208411    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4a8d39-c822-4e84-8c2d-7f57bc070439-kube-api-access-lhrc8\" (UniqueName: \"kubernetes.io/projected/cc4a8d39-c822-4e84-8c2d-7f57bc070439-kube-api-access-lhrc8\") pod \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\" (UID: \"cc4a8d39-c822-4e84-8c2d-7f57bc070439\") "
Oct 30 19:26:55.209062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.209040    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-openshift-ca" pod "cc4a8d39-c822-4e84-8c2d-7f57bc070439" (UID: "cc4a8d39-c822-4e84-8c2d-7f57bc070439"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:55.215392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.215368    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjfl\" (UniqueName: \"kubernetes.io/projected/0879ff3c-0e8f-4682-9a68-47d88140a658-kube-api-access-rvjfl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:55.215541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.215522    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:55.217358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.217326    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-xks-ca" pod "cc4a8d39-c822-4e84-8c2d-7f57bc070439" (UID: "cc4a8d39-c822-4e84-8c2d-7f57bc070439"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:55.224183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.224148    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4a8d39-c822-4e84-8c2d-7f57bc070439-kube-api-access-lhrc8" pod "cc4a8d39-c822-4e84-8c2d-7f57bc070439" (UID: "cc4a8d39-c822-4e84-8c2d-7f57bc070439"). InnerVolumeSpecName "kube-api-access-lhrc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:55.255534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.240199    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 19:26:55.255534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.240256    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 19:26:55.292084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.292037    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0879ff3c-0e8f-4682-9a68-47d88140a658" path="/var/lib/kubelet/pods/0879ff3c-0e8f-4682-9a68-47d88140a658/volumes"
Oct 30 19:26:55.331715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.323101    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrc8\" (UniqueName: \"kubernetes.io/projected/cc4a8d39-c822-4e84-8c2d-7f57bc070439-kube-api-access-lhrc8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:55.331715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.323136    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4a8d39-c822-4e84-8c2d-7f57bc070439-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:55.794480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.794391    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:55.839337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.839309    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:55.861724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.861258    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26193413-d9e4-4ec5-b532-5c3e3583f3e4-kube-api-access-99wzh\" (UniqueName: \"kubernetes.io/projected/26193413-d9e4-4ec5-b532-5c3e3583f3e4-kube-api-access-99wzh\") pod \"26193413-d9e4-4ec5-b532-5c3e3583f3e4\" (UID: \"26193413-d9e4-4ec5-b532-5c3e3583f3e4\") "
Oct 30 19:26:55.878045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.864419    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26193413-d9e4-4ec5-b532-5c3e3583f3e4-kube-api-access-99wzh" pod "26193413-d9e4-4ec5-b532-5c3e3583f3e4" (UID: "26193413-d9e4-4ec5-b532-5c3e3583f3e4"). InnerVolumeSpecName "kube-api-access-99wzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:55.981728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:55.972717    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wzh\" (UniqueName: \"kubernetes.io/projected/26193413-d9e4-4ec5-b532-5c3e3583f3e4-kube-api-access-99wzh\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:56.017873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.015607    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9" event={"ID":"cc4a8d39-c822-4e84-8c2d-7f57bc070439","Type":"ContainerDied","Data":"fa94d1cf390b44cf3acab59b9e212cfc2200e464cce65669bc6cbbb7aff2c531"}
Oct 30 19:26:56.017873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.015737    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa94d1cf390b44cf3acab59b9e212cfc2200e464cce65669bc6cbbb7aff2c531"
Oct 30 19:26:56.017873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.015831    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"
Oct 30 19:26:56.035872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.031384    4925 generic.go:356] "Generic (PLEG): container finished" podID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" containerID="2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6" exitCode=137
Oct 30 19:26:56.035872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.031617    4925 scope.go:122] "RemoveContainer" containerID="2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6"
Oct 30 19:26:56.035872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.031854    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:56.053950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.053926    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8861-795efb"
Oct 30 19:26:56.070573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.070048    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:26:56.070573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.070095    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:26:56.075240 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:56.075127    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-ca1e7f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:56.075493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.075479    4925 scope.go:122] "RemoveContainer" containerID="2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6"
Oct 30 19:26:56.078442 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:56.078421    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6\": container with ID starting with 2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6 not found: ID does not exist" containerID="2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6"
Oct 30 19:26:56.078567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.078453    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6"} err="failed to get container status \"2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6\": rpc error: code = NotFound desc = could not find container \"2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6\": container with ID starting with 2b2f36a3558da901c4b7ef2671a4a02efe81b6eb6b521ea2d4f296afd98caae6 not found: ID does not exist"
Oct 30 19:26:56.078567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.078470    4925 scope.go:122] "RemoveContainer" containerID="43a80654cff3e7e982e42335168fdecb9c6f7286610e9870821169ce28a01d9c"
Oct 30 19:26:56.079079 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:56.078801    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-ca1e7f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:56.202294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.202247    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:56.242751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.242725    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:56.244110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.244086    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:26:56.245194 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:56.245164    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-ca1e7f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:56.280744 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:56.280711    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-ca1e7f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:56.472678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.472641    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:56.570595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.570565    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"]
Oct 30 19:26:56.571256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.571238    4925 cpu_manager.go:414] "containerMap: removing container" podUID="cc4a8d39-c822-4e84-8c2d-7f57bc070439" containerName="revoke-keys"
Oct 30 19:26:56.571256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.571255    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4a8d39-c822-4e84-8c2d-7f57bc070439" containerName="revoke-keys"
Oct 30 19:26:56.571403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.571267    4925 cpu_manager.go:414] "containerMap: removing container" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" containerName="test-curl-8861-ca1e7f"
Oct 30 19:26:56.571403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.571273    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" containerName="test-curl-8861-ca1e7f"
Oct 30 19:26:56.571403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.571385    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4a8d39-c822-4e84-8c2d-7f57bc070439" containerName="revoke-keys"
Oct 30 19:26:56.571403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.571393    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" containerName="test-curl-8861-ca1e7f"
Oct 30 19:26:56.576977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.576953    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.579574 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:56.579542    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-ca1e7f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:56.586828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.586263    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b281d5-openshift-default-dockercfg-tjwsd\""
Oct 30 19:26:56.603351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.603324    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"]
Oct 30 19:26:56.689747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.689706    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-certs\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.690021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.690002    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-data\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.694877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.694843    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-497d6\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-kube-api-access-497d6\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.711713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.695059    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-istio-token\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.712093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.712066    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8313268-9549-42a2-942b-aac4e43c75a3-istiod-ca-cert\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.712354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.712337    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8313268-9549-42a2-942b-aac4e43c75a3-istio-podinfo\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.712488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.712473    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.712616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.712603    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-credential-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.712759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.712743    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-envoy\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.713008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.703733    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:56.818111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.817995    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-envoy\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.818777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.818741    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-certs\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.818987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.818955    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-data\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.819281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.819260    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-497d6\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-kube-api-access-497d6\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.819475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.819445    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-istio-token\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.820912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.820880    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8313268-9549-42a2-942b-aac4e43c75a3-istiod-ca-cert\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.821215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.821197    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8313268-9549-42a2-942b-aac4e43c75a3-istio-podinfo\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.821386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.821364    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.821525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.821511    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-credential-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.827811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.827772    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-credential-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.828397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.828376    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.828863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.828847    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-certs\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.833036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.833017    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8313268-9549-42a2-942b-aac4e43c75a3-istiod-ca-cert\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.833992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.833971    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-data\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.878414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.878144    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-envoy\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.916491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.916456    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-istio-token\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.917803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.917779    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8313268-9549-42a2-942b-aac4e43c75a3-istio-podinfo\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.940522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.936444    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-497d6\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-kube-api-access-497d6\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:56.977189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:56.977141    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:57.068202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.068096    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:26:57.069734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.069709    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:26:57.086616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.084461    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:26:57.086616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.086196    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:26:57.093341 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:57.093291    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-ca1e7f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:57.168929 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:26:57.167629    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8861-ca1e7f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" pod="opendatahub/test-curl-8861-ca1e7f"
Oct 30 19:26:57.196022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.195671    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:57.267231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.266794    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26193413-d9e4-4ec5-b532-5c3e3583f3e4" path="/var/lib/kubelet/pods/26193413-d9e4-4ec5-b532-5c3e3583f3e4/volumes"
Oct 30 19:26:57.418716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.416587    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:57.517713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.516772    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"]
Oct 30 19:26:57.529837 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:26:57.528806    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8313268_9549_42a2_942b_aac4e43c75a3.slice/crio-61945daa661c6836ade573195eac44f6988cb14a0d90b3b2119b1953132ad66b WatchSource:0}: Error finding container 61945daa661c6836ade573195eac44f6988cb14a0d90b3b2119b1953132ad66b: Status 404 returned error can't find the container with id 61945daa661c6836ade573195eac44f6988cb14a0d90b3b2119b1953132ad66b
Oct 30 19:26:57.532394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.532357    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:57.532517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.532431    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:57.532517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:57.532460    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:26:58.096903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:58.096869    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" event={"ID":"f8313268-9549-42a2-942b-aac4e43c75a3","Type":"ContainerStarted","Data":"28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1"}
Oct 30 19:26:58.097426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:58.096922    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" event={"ID":"f8313268-9549-42a2-942b-aac4e43c75a3","Type":"ContainerStarted","Data":"61945daa661c6836ade573195eac44f6988cb14a0d90b3b2119b1953132ad66b"}
Oct 30 19:26:58.130213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:58.130168    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" podStartSLOduration=2.130152777 podStartE2EDuration="2.130152777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:58.124431663 +0000 UTC m=+1253.027444694" watchObservedRunningTime="2026-10-30 19:26:58.130152777 +0000 UTC m=+1253.033166234"
Oct 30 19:26:58.196641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:58.196607    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:58.201907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:58.201882    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:58.455713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:58.452853    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:58.766907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:58.766822    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:58.976480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:58.976440    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:59.105484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:59.105397    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:59.106318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:59.106301    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:26:59.876611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:59.876576    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:59.946817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:59.946784    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"]
Oct 30 19:26:59.951031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:59.951003    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:26:59.989994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:26:59.976847    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9a718a\""
Oct 30 19:27:00.008710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.004985    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9a718a-dockercfg-mdw8c\""
Oct 30 19:27:00.088844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.088656    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"]
Oct 30 19:27:00.164124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.159857    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646ht\" (UniqueName: \"kubernetes.io/projected/24993ca9-1cac-4bff-8c7f-813c98303bf6-kube-api-access-646ht\") pod \"payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs\" (UID: \"24993ca9-1cac-4bff-8c7f-813c98303bf6\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:27:00.164124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.160055    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24993ca9-1cac-4bff-8c7f-813c98303bf6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs\" (UID: \"24993ca9-1cac-4bff-8c7f-813c98303bf6\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:27:00.210466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.210431    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:00.262398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.262366    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646ht\" (UniqueName: \"kubernetes.io/projected/24993ca9-1cac-4bff-8c7f-813c98303bf6-kube-api-access-646ht\") pod \"payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs\" (UID: \"24993ca9-1cac-4bff-8c7f-813c98303bf6\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:27:00.262603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.262566    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24993ca9-1cac-4bff-8c7f-813c98303bf6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs\" (UID: \"24993ca9-1cac-4bff-8c7f-813c98303bf6\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:27:00.263357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.263332    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24993ca9-1cac-4bff-8c7f-813c98303bf6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs\" (UID: \"24993ca9-1cac-4bff-8c7f-813c98303bf6\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:27:00.284742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.284709    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-646ht\" (UniqueName: \"kubernetes.io/projected/24993ca9-1cac-4bff-8c7f-813c98303bf6-kube-api-access-646ht\") pod \"payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs\" (UID: \"24993ca9-1cac-4bff-8c7f-813c98303bf6\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:27:00.474611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.474534    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:27:00.492232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.492188    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:00.766831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:00.766801    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"]
Oct 30 19:27:00.767765 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:27:00.767735    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24993ca9_1cac_4bff_8c7f_813c98303bf6.slice/crio-04399bc56dba4ce2f9dfbbc01645f26350eb8fa3be9426253226a479ba02e9c6 WatchSource:0}: Error finding container 04399bc56dba4ce2f9dfbbc01645f26350eb8fa3be9426253226a479ba02e9c6: Status 404 returned error can't find the container with id 04399bc56dba4ce2f9dfbbc01645f26350eb8fa3be9426253226a479ba02e9c6
Oct 30 19:27:01.036061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:01.035984    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:01.145524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:01.145490    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs" event={"ID":"24993ca9-1cac-4bff-8c7f-813c98303bf6","Type":"ContainerStarted","Data":"322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa"}
Oct 30 19:27:01.145721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:01.145572    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs" event={"ID":"24993ca9-1cac-4bff-8c7f-813c98303bf6","Type":"ContainerStarted","Data":"04399bc56dba4ce2f9dfbbc01645f26350eb8fa3be9426253226a479ba02e9c6"}
Oct 30 19:27:01.145868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:01.145805    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:27:01.177411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:01.177316    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs" podStartSLOduration=2.177298715 podStartE2EDuration="2.177298715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:01.176834834 +0000 UTC m=+1256.079847883" watchObservedRunningTime="2026-10-30 19:27:01.177298715 +0000 UTC m=+1256.080311753"
Oct 30 19:27:01.286425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:01.286349    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:02.497185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.497151    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:02.690973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.690941    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"]
Oct 30 19:27:02.695575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.695557    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:02.717219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.717195    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"]
Oct 30 19:27:02.719364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.719331    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b281d5\""
Oct 30 19:27:02.721682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.721659    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b281d5-dockercfg-2fwnk\""
Oct 30 19:27:02.736000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.735972    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"]
Oct 30 19:27:02.741571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.741551    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:02.757247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.757221    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"]
Oct 30 19:27:02.801238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.801206    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc197f-e609-4a15-a323-194cc2fbbf95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h\" (UID: \"55bc197f-e609-4a15-a323-194cc2fbbf95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:02.801422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.801341    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd9fde7-3646-4e83-a958-9aad45c10daf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk\" (UID: \"6bd9fde7-3646-4e83-a958-9aad45c10daf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:02.801517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.801501    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrz8h\" (UniqueName: \"kubernetes.io/projected/6bd9fde7-3646-4e83-a958-9aad45c10daf-kube-api-access-qrz8h\") pod \"payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk\" (UID: \"6bd9fde7-3646-4e83-a958-9aad45c10daf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:02.801583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.801569    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxvp\" (UniqueName: \"kubernetes.io/projected/55bc197f-e609-4a15-a323-194cc2fbbf95-kube-api-access-phxvp\") pod \"payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h\" (UID: \"55bc197f-e609-4a15-a323-194cc2fbbf95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:02.809939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.809914    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:02.910639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.906772    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc197f-e609-4a15-a323-194cc2fbbf95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h\" (UID: \"55bc197f-e609-4a15-a323-194cc2fbbf95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:02.910639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.906897    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd9fde7-3646-4e83-a958-9aad45c10daf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk\" (UID: \"6bd9fde7-3646-4e83-a958-9aad45c10daf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:02.910639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.907106    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrz8h\" (UniqueName: \"kubernetes.io/projected/6bd9fde7-3646-4e83-a958-9aad45c10daf-kube-api-access-qrz8h\") pod \"payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk\" (UID: \"6bd9fde7-3646-4e83-a958-9aad45c10daf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:02.910639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.907203    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxvp\" (UniqueName: \"kubernetes.io/projected/55bc197f-e609-4a15-a323-194cc2fbbf95-kube-api-access-phxvp\") pod \"payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h\" (UID: \"55bc197f-e609-4a15-a323-194cc2fbbf95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:02.935719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.934638    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd9fde7-3646-4e83-a958-9aad45c10daf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk\" (UID: \"6bd9fde7-3646-4e83-a958-9aad45c10daf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:02.943715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.941102    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc197f-e609-4a15-a323-194cc2fbbf95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h\" (UID: \"55bc197f-e609-4a15-a323-194cc2fbbf95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:02.967670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:02.967640    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxvp\" (UniqueName: \"kubernetes.io/projected/55bc197f-e609-4a15-a323-194cc2fbbf95-kube-api-access-phxvp\") pod \"payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h\" (UID: \"55bc197f-e609-4a15-a323-194cc2fbbf95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:03.014113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:03.014028    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:03.018819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:03.018792    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrz8h\" (UniqueName: \"kubernetes.io/projected/6bd9fde7-3646-4e83-a958-9aad45c10daf-kube-api-access-qrz8h\") pod \"payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk\" (UID: \"6bd9fde7-3646-4e83-a958-9aad45c10daf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:03.067735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:03.067156    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:03.067735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:03.067473    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:03.419338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:03.419307    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"]
Oct 30 19:27:03.420501 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:27:03.420478    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bc197f_e609_4a15_a323_194cc2fbbf95.slice/crio-7d1fe3077b0f5485d372e6fe79e06c8b4242674f43233d0587ad2c7e6d62a73a WatchSource:0}: Error finding container 7d1fe3077b0f5485d372e6fe79e06c8b4242674f43233d0587ad2c7e6d62a73a: Status 404 returned error can't find the container with id 7d1fe3077b0f5485d372e6fe79e06c8b4242674f43233d0587ad2c7e6d62a73a
Oct 30 19:27:03.436862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:03.436837    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"]
Oct 30 19:27:03.436953 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:27:03.436880    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd9fde7_3646_4e83_a958_9aad45c10daf.slice/crio-8c9bad36468e2d3e086873528637c9513de544e98e10d902358d82d24eef55fa WatchSource:0}: Error finding container 8c9bad36468e2d3e086873528637c9513de544e98e10d902358d82d24eef55fa: Status 404 returned error can't find the container with id 8c9bad36468e2d3e086873528637c9513de544e98e10d902358d82d24eef55fa
Oct 30 19:27:04.138470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:04.138442    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:04.177938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:04.177887    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h" event={"ID":"55bc197f-e609-4a15-a323-194cc2fbbf95","Type":"ContainerStarted","Data":"ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f"}
Oct 30 19:27:04.178121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:04.178000    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h" event={"ID":"55bc197f-e609-4a15-a323-194cc2fbbf95","Type":"ContainerStarted","Data":"7d1fe3077b0f5485d372e6fe79e06c8b4242674f43233d0587ad2c7e6d62a73a"}
Oct 30 19:27:04.178250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:04.178166    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:04.180083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:04.180057    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk" event={"ID":"6bd9fde7-3646-4e83-a958-9aad45c10daf","Type":"ContainerStarted","Data":"60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78"}
Oct 30 19:27:04.180288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:04.180262    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk" event={"ID":"6bd9fde7-3646-4e83-a958-9aad45c10daf","Type":"ContainerStarted","Data":"8c9bad36468e2d3e086873528637c9513de544e98e10d902358d82d24eef55fa"}
Oct 30 19:27:04.180386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:04.180308    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:04.219467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:04.219405    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h" podStartSLOduration=2.219385948 podStartE2EDuration="2.219385948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:04.215755624 +0000 UTC m=+1259.118768661" watchObservedRunningTime="2026-10-30 19:27:04.219385948 +0000 UTC m=+1259.122398990"
Oct 30 19:27:04.239558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:04.238649    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk" podStartSLOduration=2.238628543 podStartE2EDuration="2.238628543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:04.238153378 +0000 UTC m=+1259.141166417" watchObservedRunningTime="2026-10-30 19:27:04.238628543 +0000 UTC m=+1259.141641582"
Oct 30 19:27:05.039001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:05.038966    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"]
Oct 30 19:27:05.047429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:05.047401    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa3296ac-xq4nd"]
Oct 30 19:27:05.273463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:05.273433    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a4d7df-1431-44a1-8718-98a44bd9561c" path="/var/lib/kubelet/pods/26a4d7df-1431-44a1-8718-98a44bd9561c/volumes"
Oct 30 19:27:05.401449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:05.394432    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:05.939717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:05.939076    4925 scope.go:122] "RemoveContainer" containerID="2d1f6e787fbcd42eb5897f65a6cfdb9206eb21526dbf424fa2ed2e412f44a93c"
Oct 30 19:27:05.983976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:05.983953    4925 scope.go:122] "RemoveContainer" containerID="dd9108a45c469736a07ed524e20933e2b0fd5bf578a890c19f43779a1a124eec"
Oct 30 19:27:05.993996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:05.993975    4925 scope.go:122] "RemoveContainer" containerID="5bb3527282f0c55def48bed3b1ff1381f251e68e06eaf49acbbc7bb869dbb6a6"
Oct 30 19:27:06.077425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:06.077393    4925 scope.go:122] "RemoveContainer" containerID="ff88b1c5b1cc4d63923d790c1ebf281039b4814cca80f107b6e2a431f424703c"
Oct 30 19:27:06.418759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:06.412063    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:07.403826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:07.403770    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:07.936050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:07.936020    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:08.393199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:08.393146    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:09.159494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:09.159460    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:09.995046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:09.995018    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:10.862924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:10.862253    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:11.957520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:11.957488    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:12.156171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:12.156144    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:27:13.002470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.002385    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:13.471839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.471806    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:13.767792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.767707    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"]
Oct 30 19:27:13.773975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.773952    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.778779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.778754    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9a718a-serving-cert\""
Oct 30 19:27:13.792280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.792260    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"]
Oct 30 19:27:13.840886 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.840807    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-metrics-certs\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.852022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.851983    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7zc2\" (UniqueName: \"kubernetes.io/projected/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-kube-api-access-v7zc2\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.852234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.852095    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-maas-api-tls\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.852234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.852142    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-openshift-service-ca\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.949374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.949342    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:13.953838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.953809    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-maas-api-tls\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.954014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.953857    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-openshift-service-ca\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.954014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.953993    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-metrics-certs\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.954166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.954151    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7zc2\" (UniqueName: \"kubernetes.io/projected/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-kube-api-access-v7zc2\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.954731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.954706    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-openshift-service-ca\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.956874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.956847    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-metrics-certs\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.957013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.956849    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-maas-api-tls\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:13.971678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:13.971646    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7zc2\" (UniqueName: \"kubernetes.io/projected/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-kube-api-access-v7zc2\") pod \"maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:14.032356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:14.032280    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"]
Oct 30 19:27:14.037460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:14.037436    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb9f8cc-hj9fw"]
Oct 30 19:27:14.088109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:14.088082    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:14.133265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:14.133234    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:14.379635 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:27:14.379604    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf4cf20_f9a7_4ff1_ae2e_7741b49fddd3.slice/crio-316524c853b2a5111f769a9357dfff53a1b66547128947e3cfd1af42b23b7aee WatchSource:0}: Error finding container 316524c853b2a5111f769a9357dfff53a1b66547128947e3cfd1af42b23b7aee: Status 404 returned error can't find the container with id 316524c853b2a5111f769a9357dfff53a1b66547128947e3cfd1af42b23b7aee
Oct 30 19:27:14.379798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:14.379653    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"]
Oct 30 19:27:15.194484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.194405    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:27:15.195041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.194491    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:27:15.241557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.241525    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"]
Oct 30 19:27:15.246588 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.246565    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.264755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.262490    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-48b1a6a7\"/\"default-dockercfg-fcwbx\""
Oct 30 19:27:15.264755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.262660    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-48b1a6a7\"/\"e2e-lifecycle-model-48b1a6a7-kserve-self-signed-certs\""
Oct 30 19:27:15.264755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.263505    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-48b1a6a7\"/\"kube-root-ca.crt\""
Oct 30 19:27:15.265442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.265306    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-48b1a6a7\"/\"openshift-service-ca.crt\""
Oct 30 19:27:15.273923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.273899    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40add373-e5a0-42f5-ba37-2b578dedba7f" path="/var/lib/kubelet/pods/40add373-e5a0-42f5-ba37-2b578dedba7f/volumes"
Oct 30 19:27:15.275035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.275014    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"]
Oct 30 19:27:15.339188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.339152    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:15.383602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.383574    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7" event={"ID":"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3","Type":"ContainerStarted","Data":"3ef3ad607dba6e21b248c93ba67c0f7041e4b43f5ac1a88e56e94505ff7915d5"}
Oct 30 19:27:15.383847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.383832    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7" event={"ID":"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3","Type":"ContainerStarted","Data":"316524c853b2a5111f769a9357dfff53a1b66547128947e3cfd1af42b23b7aee"}
Oct 30 19:27:15.384592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.384574    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:15.404137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.404104    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafe5212-0e46-4f23-b375-848434828400-tls-certs\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.404535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.404509    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9kk6\" (UniqueName: \"kubernetes.io/projected/eafe5212-0e46-4f23-b375-848434828400-kube-api-access-g9kk6\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.404732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.404713    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-home\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.404845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.404764    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-model-cache\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.404845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.404817    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-dshm\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.404951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.404931    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-tmp-dir\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.507767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.502554    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7" podStartSLOduration=1.9755090709999998 podStartE2EDuration="2.502528421s" totalImagesPullingTime="527.01935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:15.456105413 +0000 UTC m=+1270.359118451" watchObservedRunningTime="2026-10-30 19:27:15.502528421 +0000 UTC m=+1270.405541462"
Oct 30 19:27:15.513713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.508207    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-dshm\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.513713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.508777    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-tmp-dir\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.513713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.508930    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafe5212-0e46-4f23-b375-848434828400-tls-certs\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.513713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.509224    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9kk6\" (UniqueName: \"kubernetes.io/projected/eafe5212-0e46-4f23-b375-848434828400-kube-api-access-g9kk6\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.513713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.509309    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-home\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.513713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.509364    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-model-cache\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.513713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.509820    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-model-cache\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.513713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.510063    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-tmp-dir\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.522042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.518591    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-home\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.563709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.561411    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafe5212-0e46-4f23-b375-848434828400-tls-certs\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.563709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.561411    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9kk6\" (UniqueName: \"kubernetes.io/projected/eafe5212-0e46-4f23-b375-848434828400-kube-api-access-g9kk6\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.575780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.574630    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-dshm\") pod \"e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") " pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.593718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.592198    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:15.603390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.603094    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:15.995663 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:27:15.995633    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeafe5212_0e46_4f23_b375_848434828400.slice/crio-f635afb80c6006727840dd7f61e3e42ce0a47d2159b3728e9479ca65c5f0dff0 WatchSource:0}: Error finding container f635afb80c6006727840dd7f61e3e42ce0a47d2159b3728e9479ca65c5f0dff0: Status 404 returned error can't find the container with id f635afb80c6006727840dd7f61e3e42ce0a47d2159b3728e9479ca65c5f0dff0
Oct 30 19:27:15.997517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:15.997490    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"]
Oct 30 19:27:16.236149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:16.236111    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:16.394907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:16.394870    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" event={"ID":"eafe5212-0e46-4f23-b375-848434828400","Type":"ContainerStarted","Data":"a638b417eed1cc7f0bcd0f72a335aede6aae3f108956401a7388fbed888a83f7"}
Oct 30 19:27:16.395060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:16.394921    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" event={"ID":"eafe5212-0e46-4f23-b375-848434828400","Type":"ContainerStarted","Data":"f635afb80c6006727840dd7f61e3e42ce0a47d2159b3728e9479ca65c5f0dff0"}
Oct 30 19:27:16.420002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:16.419950    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" podStartSLOduration=1.419936256 podStartE2EDuration="1.419936256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:16.415164804 +0000 UTC m=+1271.318177841" watchObservedRunningTime="2026-10-30 19:27:16.419936256 +0000 UTC m=+1271.322949293"
Oct 30 19:27:16.604644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:16.604609    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:17.103861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:17.103823    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:17.308133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:17.308099    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:18.126594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:18.126561    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:19.006838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:19.006807    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:19.716400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:19.716366    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:20.652323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:20.652285    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:21.332868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:21.332248    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:22.017508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:22.017473    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:22.133986 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:27:22.133910    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:27:22.410650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:22.410618    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:27:22.859358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:22.859271    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:23.474929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:23.474888    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:24.160837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:24.160807    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:25.056129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:25.056091    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:25.604753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:25.604718    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:25.617785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:25.617762    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:25.745865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:25.745828    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:26.471870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:26.471838    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:26.541398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:26.540542    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:27:27.303190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:27.303157    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:28.198132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:28.198097    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:28.997645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:28.993266    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:29.921562 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:29.921531    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:30.380442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:30.380345    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:30.603490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:30.603457    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:30.954552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:30.954507    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:31.287779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:31.281943    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:31.614920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:31.614834    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:31.794642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:31.794609    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:32.031227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.031183    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"]
Oct 30 19:27:32.036372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.036350    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1064da-zvcw6"]
Oct 30 19:27:32.081555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.081524    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:32.291816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.291728    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:32.638794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.638613    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"]
Oct 30 19:27:32.659426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.659400    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"]
Oct 30 19:27:32.659768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.659751    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.719061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.719035    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-8f3ff4-openshift-default-dockercfg-jq5kg\""
Oct 30 19:27:32.770713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.770477    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:32.860925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.860887    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-podinfo\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.861183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.861162    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-token\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.861308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.861291    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-credential-socket\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.861456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.861442    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-certs\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.861581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.861566    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-data\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.861753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.861721    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-socket\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.861935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.861917    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-envoy\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.862232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.862213    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9fnk\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-kube-api-access-g9fnk\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.862420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.862407    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istiod-ca-cert\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.965163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.965099    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9fnk\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-kube-api-access-g9fnk\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.965632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.965606    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istiod-ca-cert\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.965791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.965762    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-podinfo\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.965857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.965816    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-token\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.965930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.965857    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-credential-socket\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.965930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.965902    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-certs\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.967643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.967601    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istiod-ca-cert\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.968374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.968334    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-certs\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.969102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.969080    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-credential-socket\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.969343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.969326    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-data\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.969777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.969759    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-socket\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.976664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.976642    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-envoy\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.977417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.970132    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-socket\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.977566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.969661    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-data\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.995095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.991994    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-envoy\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.996229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.996207    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-podinfo\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:32.998390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:32.998339    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9fnk\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-kube-api-access-g9fnk\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:33.038331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.038253    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-token\") pod \"e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") " pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:33.147447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.147291    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:33.265040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.264967    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e96f89d-f177-43b9-847b-cc7caca2250d" path="/var/lib/kubelet/pods/4e96f89d-f177-43b9-847b-cc7caca2250d/volumes"
Oct 30 19:27:33.289958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.289929    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:33.600906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.600875    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"]
Oct 30 19:27:33.602787 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:27:33.602663    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4afab18_0b83_4db6_a92e_3ad1c5a50da9.slice/crio-af7d223ff90bd27e0d013b6ea3364dc511204e8560c4940c5cbd5af0b7d16e1c WatchSource:0}: Error finding container af7d223ff90bd27e0d013b6ea3364dc511204e8560c4940c5cbd5af0b7d16e1c: Status 404 returned error can't find the container with id af7d223ff90bd27e0d013b6ea3364dc511204e8560c4940c5cbd5af0b7d16e1c
Oct 30 19:27:33.605988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.605955    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:27:33.606065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.606023    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:27:33.606065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.606052    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:27:33.666487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.666438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:33.711535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:33.711288    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" event={"ID":"b4afab18-0b83-4db6-a92e-3ad1c5a50da9","Type":"ContainerStarted","Data":"af7d223ff90bd27e0d013b6ea3364dc511204e8560c4940c5cbd5af0b7d16e1c"}
Oct 30 19:27:34.723913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:34.723876    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" event={"ID":"b4afab18-0b83-4db6-a92e-3ad1c5a50da9","Type":"ContainerStarted","Data":"e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd"}
Oct 30 19:27:34.766475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:34.766428    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" podStartSLOduration=2.766413209 podStartE2EDuration="2.766413209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:34.765392672 +0000 UTC m=+1289.668405709" watchObservedRunningTime="2026-10-30 19:27:34.766413209 +0000 UTC m=+1289.669426246"
Oct 30 19:27:35.001817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:34.997968    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:35.242978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:35.242945    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:35.290285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:35.290197    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:35.301020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:35.300993    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:35.443776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:35.443745    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:35.743081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:35.743053    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:35.744232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:35.744197    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:27:36.033508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.033419    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:36.780342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.780301    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:36.827336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.827302    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"]
Oct 30 19:27:36.834218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.834192    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:36.835416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.835395    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"]
Oct 30 19:27:36.839643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.839625    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:36.846897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.846876    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9b2e27\""
Oct 30 19:27:36.846897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.846881    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9b2e27-dockercfg-fkqg4\""
Oct 30 19:27:36.847232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.847213    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"]
Oct 30 19:27:36.853680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.853661    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"]
Oct 30 19:27:36.886108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.886071    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78pb\" (UniqueName: \"kubernetes.io/projected/8c0a7656-3a0d-4cd4-a250-80218d696f48-kube-api-access-n78pb\") pod \"payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc\" (UID: \"8c0a7656-3a0d-4cd4-a250-80218d696f48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:36.886272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.886222    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effed5a1-a05b-4955-aca9-3ca8a882c094-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5\" (UID: \"effed5a1-a05b-4955-aca9-3ca8a882c094\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:36.886361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.886284    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zl9f\" (UniqueName: \"kubernetes.io/projected/effed5a1-a05b-4955-aca9-3ca8a882c094-kube-api-access-8zl9f\") pod \"payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5\" (UID: \"effed5a1-a05b-4955-aca9-3ca8a882c094\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:36.886436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:36.886419    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0a7656-3a0d-4cd4-a250-80218d696f48-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc\" (UID: \"8c0a7656-3a0d-4cd4-a250-80218d696f48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:37.096580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.096484    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effed5a1-a05b-4955-aca9-3ca8a882c094-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5\" (UID: \"effed5a1-a05b-4955-aca9-3ca8a882c094\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:37.096925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.096902    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zl9f\" (UniqueName: \"kubernetes.io/projected/effed5a1-a05b-4955-aca9-3ca8a882c094-kube-api-access-8zl9f\") pod \"payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5\" (UID: \"effed5a1-a05b-4955-aca9-3ca8a882c094\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:37.097169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.097150    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0a7656-3a0d-4cd4-a250-80218d696f48-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc\" (UID: \"8c0a7656-3a0d-4cd4-a250-80218d696f48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:37.097541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.097523    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78pb\" (UniqueName: \"kubernetes.io/projected/8c0a7656-3a0d-4cd4-a250-80218d696f48-kube-api-access-n78pb\") pod \"payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc\" (UID: \"8c0a7656-3a0d-4cd4-a250-80218d696f48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:37.159729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.099462    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effed5a1-a05b-4955-aca9-3ca8a882c094-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5\" (UID: \"effed5a1-a05b-4955-aca9-3ca8a882c094\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:37.159981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.100439    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0a7656-3a0d-4cd4-a250-80218d696f48-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc\" (UID: \"8c0a7656-3a0d-4cd4-a250-80218d696f48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:37.160111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.125580    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:37.235443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.235373    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zl9f\" (UniqueName: \"kubernetes.io/projected/effed5a1-a05b-4955-aca9-3ca8a882c094-kube-api-access-8zl9f\") pod \"payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5\" (UID: \"effed5a1-a05b-4955-aca9-3ca8a882c094\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:37.244103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.244067    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78pb\" (UniqueName: \"kubernetes.io/projected/8c0a7656-3a0d-4cd4-a250-80218d696f48-kube-api-access-n78pb\") pod \"payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc\" (UID: \"8c0a7656-3a0d-4cd4-a250-80218d696f48\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:37.457321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.457289    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:37.465532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.465491    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:37.685020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.684550    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:37.974789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.974754    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:37.986920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:37.986897    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"]
Oct 30 19:27:37.988930 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:27:37.988902    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0a7656_3a0d_4cd4_a250_80218d696f48.slice/crio-b308b85edb3b143d71c21a0ad928894edbe547dc991f17a645e35e8eb462ade0 WatchSource:0}: Error finding container b308b85edb3b143d71c21a0ad928894edbe547dc991f17a645e35e8eb462ade0: Status 404 returned error can't find the container with id b308b85edb3b143d71c21a0ad928894edbe547dc991f17a645e35e8eb462ade0
Oct 30 19:27:38.033057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.030814    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"]
Oct 30 19:27:38.033057 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:27:38.031956    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffed5a1_a05b_4955_aca9_3ca8a882c094.slice/crio-4969bdbc852bd282a763492a59a40890f7e6e2231797a757e4a5f84bde6ae33a WatchSource:0}: Error finding container 4969bdbc852bd282a763492a59a40890f7e6e2231797a757e4a5f84bde6ae33a: Status 404 returned error can't find the container with id 4969bdbc852bd282a763492a59a40890f7e6e2231797a757e4a5f84bde6ae33a
Oct 30 19:27:38.376622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.376543    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:38.777871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.777841    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc" event={"ID":"8c0a7656-3a0d-4cd4-a250-80218d696f48","Type":"ContainerStarted","Data":"8a59e214d3afaae0e4408fde016bf3dd690f280af85b4bab6070121f5862c6de"}
Oct 30 19:27:38.778073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.777890    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc" event={"ID":"8c0a7656-3a0d-4cd4-a250-80218d696f48","Type":"ContainerStarted","Data":"b308b85edb3b143d71c21a0ad928894edbe547dc991f17a645e35e8eb462ade0"}
Oct 30 19:27:38.778073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.778026    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:38.779560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.779535    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5" event={"ID":"effed5a1-a05b-4955-aca9-3ca8a882c094","Type":"ContainerStarted","Data":"77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae"}
Oct 30 19:27:38.779719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.779575    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5" event={"ID":"effed5a1-a05b-4955-aca9-3ca8a882c094","Type":"ContainerStarted","Data":"4969bdbc852bd282a763492a59a40890f7e6e2231797a757e4a5f84bde6ae33a"}
Oct 30 19:27:38.779719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.779596    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:38.790330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.790305    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:38.846346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.841891    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc" podStartSLOduration=2.84187522 podStartE2EDuration="2.84187522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:38.829016342 +0000 UTC m=+1293.732029382" watchObservedRunningTime="2026-10-30 19:27:38.84187522 +0000 UTC m=+1293.744888256"
Oct 30 19:27:38.854988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:38.854942    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5" podStartSLOduration=2.854922448 podStartE2EDuration="2.854922448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:38.852748066 +0000 UTC m=+1293.755761103" watchObservedRunningTime="2026-10-30 19:27:38.854922448 +0000 UTC m=+1293.757935487"
Oct 30 19:27:39.321277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:39.321163    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:39.559711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:39.551047    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:40.061374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:40.061338    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:41.072586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.072542    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:41.705340 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.705306    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"]
Oct 30 19:27:41.711219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.711195    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.732517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.732486    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"]
Oct 30 19:27:41.736422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.736389    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-data\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.736589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.736514    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-certs\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.736656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.736597    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.736948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.736925    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-credential-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.737088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.737026    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-token\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.737088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.737067    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a871e328-d34b-40c8-a5b0-55b97d00dd34-istiod-ca-cert\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.737200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.737093    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-envoy\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.737200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.737121    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46qq4\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-kube-api-access-46qq4\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.737200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.737152    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-podinfo\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.838656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.838626    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.838884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.838814    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-credential-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.838884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.838868    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-token\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.838890    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a871e328-d34b-40c8-a5b0-55b97d00dd34-istiod-ca-cert\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.838914    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-envoy\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.838933    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qq4\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-kube-api-access-46qq4\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.838955    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-podinfo\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.838998    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-data\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.839065    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-certs\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.839156    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.839431    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-certs\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.839509    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-credential-socket\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.839529    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-data\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.839937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.839746    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a871e328-d34b-40c8-a5b0-55b97d00dd34-istiod-ca-cert\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.841468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.841445    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-envoy\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.841664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.841649    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-podinfo\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.851276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.851250    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-token\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.851403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.851330    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qq4\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-kube-api-access-46qq4\") pod \"e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") " pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:41.889169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:41.889088    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:42.038259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:42.038186    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:42.353626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:42.353594    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"]
Oct 30 19:27:42.357919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:42.357878    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:27:42.358022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:42.357969    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:27:42.358022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:42.358013    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:27:42.672713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:42.668142    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:42.831099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:42.831059    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" event={"ID":"a871e328-d34b-40c8-a5b0-55b97d00dd34","Type":"ContainerStarted","Data":"2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d"}
Oct 30 19:27:42.831284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:42.831123    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" event={"ID":"a871e328-d34b-40c8-a5b0-55b97d00dd34","Type":"ContainerStarted","Data":"1342afec32fa838e6ec18094243901748cf494316d29d329d0bdd7736a709b13"}
Oct 30 19:27:42.858748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:42.858679    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" podStartSLOduration=1.858664402 podStartE2EDuration="1.858664402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:42.856781183 +0000 UTC m=+1297.759794220" watchObservedRunningTime="2026-10-30 19:27:42.858664402 +0000 UTC m=+1297.761677439"
Oct 30 19:27:43.038377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:43.038291    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:43.040337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:43.040309    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:27:43.040463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:43.040363    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:27:43.674834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:43.674797    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:44.039401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:44.039318    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:27:44.039401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:44.039378    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:27:44.392477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:44.392437    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:45.038703 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.038657    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:27:45.039134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.038734    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:27:45.085007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.084973    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:45.158892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.158842    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:27:45.159039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.158898    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:45.858291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.858204    4925 generic.go:356] "Generic (PLEG): container finished" podID="f8313268-9549-42a2-942b-aac4e43c75a3" containerID="28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1" exitCode=0
Oct 30 19:27:45.858488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.858287    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" event={"ID":"f8313268-9549-42a2-942b-aac4e43c75a3","Type":"ContainerDied","Data":"28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1"}
Oct 30 19:27:45.858868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.858846    4925 scope.go:122] "RemoveContainer" containerID="28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1"
Oct 30 19:27:45.860711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.860665    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:27:45.860823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.860792    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:27:45.860880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.860838    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:27:45.974956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:45.974918    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:46.078985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:46.065205    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:27:46.078985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:46.065265    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:27:46.923799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:46.922225    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:46.935483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:46.927815    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" event={"ID":"f8313268-9549-42a2-942b-aac4e43c75a3","Type":"ContainerStarted","Data":"e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3"}
Oct 30 19:27:47.042711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:47.042024    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:27:47.042711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:47.042078    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:27:47.205779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:47.198984    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:27:47.205779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:47.199092    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:27:47.205779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:47.199132    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:27:47.671711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:47.669743    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:48.047881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:48.047789    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:27:48.048135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:48.048106    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:27:48.207714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:48.203998    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:27:48.207714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:48.204050    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:27:48.728713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:48.728483    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:49.073565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:49.073482    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:49.074145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:49.074122    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:49.076175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:49.076151    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:27:49.213714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:49.199294    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:27:49.213714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:49.199355    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:27:49.532728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:49.532293    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:49.793335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:49.793235    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:27:49.793486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:49.793352    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:27:50.220084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:50.220049    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:27:50.220675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:50.220100    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:27:50.220675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:50.220483    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:50.423709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:50.423307    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:50.620214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:50.620123    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:51.199088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:51.199053    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:27:51.199379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:51.199349    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:27:51.343073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:51.343041    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:52.062861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:52.062830    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:52.153787 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:27:52.153744    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:27:52.196888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:52.196832    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 19:27:52.197113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:52.196952    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 19:27:52.850261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:52.850225    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:53.021638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:53.021596    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:53.036435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:53.036390    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"]
Oct 30 19:27:53.043171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:53.043140    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81cc5519-q6k5z"]
Oct 30 19:27:53.213801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:53.213624    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:27:53.214338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:53.214313    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:27:53.215296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:53.215279    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:27:53.295677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:53.295645    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6" path="/var/lib/kubelet/pods/e5288c3d-7d94-4ab4-8bb5-2d0c162b80c6/volumes"
Oct 30 19:27:53.743526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:53.743480    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:54.480487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:54.480446    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:55.152231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:55.152190    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:56.038408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:56.038372    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:56.546630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:56.546587    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:56.802775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:56.802658    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:57.255122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:57.255087    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:57.941711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:57.941532    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:58.764852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:58.764796    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:59.035721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:59.035376    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"]
Oct 30 19:27:59.040207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:59.040177    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79cf21af-h6sg4"]
Oct 30 19:27:59.264462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:59.264430    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8830d16d-a8d9-44ac-be97-c98a0e86f421" path="/var/lib/kubelet/pods/8830d16d-a8d9-44ac-be97-c98a0e86f421/volumes"
Oct 30 19:27:59.622169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:27:59.622119    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:00.438271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.438225    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:00.549794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.549766    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"]
Oct 30 19:28:00.555502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.555485    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.559235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.559195    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:28:00.559235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.559237    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:00.564801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.564759    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"]
Oct 30 19:28:00.644508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.644475    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d360546-dwfkw\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.644679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.644527    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8l8\" (UniqueName: \"kubernetes.io/projected/3336a061-d04a-4b02-914d-42941df3d8a1-kube-api-access-nx8l8\") pod \"maas-api-revoke-keys-e2e-mt-0d360546-dwfkw\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.644679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.644612    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d360546-dwfkw\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.747128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.747002    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d360546-dwfkw\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.747128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.747076    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8l8\" (UniqueName: \"kubernetes.io/projected/3336a061-d04a-4b02-914d-42941df3d8a1-kube-api-access-nx8l8\") pod \"maas-api-revoke-keys-e2e-mt-0d360546-dwfkw\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.747472 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.747126    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d360546-dwfkw\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.747872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.747833    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d360546-dwfkw\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.750554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.750527    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d360546-dwfkw\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.758659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.758634    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8l8\" (UniqueName: \"kubernetes.io/projected/3336a061-d04a-4b02-914d-42941df3d8a1-kube-api-access-nx8l8\") pod \"maas-api-revoke-keys-e2e-mt-0d360546-dwfkw\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:00.875468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:00.875437    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:01.170047 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:01.170010    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3336a061_d04a_4b02_914d_42941df3d8a1.slice/crio-633566d4dbe71f6ee21c60b20986048ca0232ebe6fe44199458bf3046b050ea9 WatchSource:0}: Error finding container 633566d4dbe71f6ee21c60b20986048ca0232ebe6fe44199458bf3046b050ea9: Status 404 returned error can't find the container with id 633566d4dbe71f6ee21c60b20986048ca0232ebe6fe44199458bf3046b050ea9
Oct 30 19:28:01.170486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:01.170467    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"]
Oct 30 19:28:01.430836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:01.430804    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:02.155902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:02.153881    4925 generic.go:356] "Generic (PLEG): container finished" podID="3336a061-d04a-4b02-914d-42941df3d8a1" containerID="f76764b02cc12ef4abfd13f1e6256b034c9ee3a2f732287ffa65f12abacdb391" exitCode=0
Oct 30 19:28:02.155902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:02.154048    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw" event={"ID":"3336a061-d04a-4b02-914d-42941df3d8a1","Type":"ContainerDied","Data":"f76764b02cc12ef4abfd13f1e6256b034c9ee3a2f732287ffa65f12abacdb391"}
Oct 30 19:28:02.155902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:02.154116    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw" event={"ID":"3336a061-d04a-4b02-914d-42941df3d8a1","Type":"ContainerStarted","Data":"633566d4dbe71f6ee21c60b20986048ca0232ebe6fe44199458bf3046b050ea9"}
Oct 30 19:28:02.158762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:02.156820    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:02.761718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:02.760479    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:03.349866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.349842    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:03.492989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.492913    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-xks-ca\") pod \"3336a061-d04a-4b02-914d-42941df3d8a1\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") "
Oct 30 19:28:03.493131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.493002    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-openshift-ca\") pod \"3336a061-d04a-4b02-914d-42941df3d8a1\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") "
Oct 30 19:28:03.493319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.493295    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3336a061-d04a-4b02-914d-42941df3d8a1-kube-api-access-nx8l8\" (UniqueName: \"kubernetes.io/projected/3336a061-d04a-4b02-914d-42941df3d8a1-kube-api-access-nx8l8\") pod \"3336a061-d04a-4b02-914d-42941df3d8a1\" (UID: \"3336a061-d04a-4b02-914d-42941df3d8a1\") "
Oct 30 19:28:03.493440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.493328    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-openshift-ca" pod "3336a061-d04a-4b02-914d-42941df3d8a1" (UID: "3336a061-d04a-4b02-914d-42941df3d8a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:03.493948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.493928    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:03.495365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.495341    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-xks-ca" pod "3336a061-d04a-4b02-914d-42941df3d8a1" (UID: "3336a061-d04a-4b02-914d-42941df3d8a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:03.495643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.495618    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3336a061-d04a-4b02-914d-42941df3d8a1-kube-api-access-nx8l8" pod "3336a061-d04a-4b02-914d-42941df3d8a1" (UID: "3336a061-d04a-4b02-914d-42941df3d8a1"). InnerVolumeSpecName "kube-api-access-nx8l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:03.551655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.551502    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:03.595607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.595577    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx8l8\" (UniqueName: \"kubernetes.io/projected/3336a061-d04a-4b02-914d-42941df3d8a1-kube-api-access-nx8l8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:03.595607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:03.595607    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3336a061-d04a-4b02-914d-42941df3d8a1-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:04.171365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:04.171335    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"
Oct 30 19:28:04.171365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:04.171352    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw" event={"ID":"3336a061-d04a-4b02-914d-42941df3d8a1","Type":"ContainerDied","Data":"633566d4dbe71f6ee21c60b20986048ca0232ebe6fe44199458bf3046b050ea9"}
Oct 30 19:28:04.171587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:04.171424    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="633566d4dbe71f6ee21c60b20986048ca0232ebe6fe44199458bf3046b050ea9"
Oct 30 19:28:04.311646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:04.311604    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:05.245792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:05.245756    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:05.704345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:05.704307    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:06.269799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:06.269753    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:06.520484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:06.520407    4925 scope.go:122] "RemoveContainer" containerID="31e45d6822162f03eebdcf060a593dbb7c20b813312f9afae62c92884962daa7"
Oct 30 19:28:06.532240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:06.532211    4925 scope.go:122] "RemoveContainer" containerID="d9ef8f481dc431a41791ebe2a08951f105265656b8046f33571d9d28cbd5312b"
Oct 30 19:28:06.542572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:06.542554    4925 scope.go:122] "RemoveContainer" containerID="1d4eb6c8d0658cd22dad3c90f721669ab1a4fed3620086c207ef8e4589d51c37"
Oct 30 19:28:07.077241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:07.077211    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:07.947937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:07.947906    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:08.644680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.644647    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"]
Oct 30 19:28:08.645460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.645437    4925 cpu_manager.go:414] "containerMap: removing container" podUID="3336a061-d04a-4b02-914d-42941df3d8a1" containerName="revoke-keys"
Oct 30 19:28:08.645460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.645460    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3336a061-d04a-4b02-914d-42941df3d8a1" containerName="revoke-keys"
Oct 30 19:28:08.645675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.645661    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3336a061-d04a-4b02-914d-42941df3d8a1" containerName="revoke-keys"
Oct 30 19:28:08.651429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.651408    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.656736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.656717    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:08.656854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.656839    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:28:08.666642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.666618    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"]
Oct 30 19:28:08.737357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.737327    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.737514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.737486    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.737556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.737529    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzpgz\" (UniqueName: \"kubernetes.io/projected/eb39302f-43cf-420b-a305-73813e446ec0-kube-api-access-fzpgz\") pod \"maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.838960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.838925    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.839121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.839024    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.839121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.839057    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpgz\" (UniqueName: \"kubernetes.io/projected/eb39302f-43cf-420b-a305-73813e446ec0-kube-api-access-fzpgz\") pod \"maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.839651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.839631    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.841496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.841477    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.850823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.850799    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpgz\" (UniqueName: \"kubernetes.io/projected/eb39302f-43cf-420b-a305-73813e446ec0-kube-api-access-fzpgz\") pod \"maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:08.967248 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:08.967181    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:09.035505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.035474    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:09.210715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.210227    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:09.407456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.407427    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"]
Oct 30 19:28:09.607734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.605308    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"]
Oct 30 19:28:09.617740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.616247    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.638927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.638113    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8ceda0-openshift-default-dockercfg-hwk6b\""
Oct 30 19:28:09.665035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.665009    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"]
Oct 30 19:28:09.678290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.678258    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istiod-ca-cert\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.678442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.678314    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-data\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.678501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.678475    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbc8\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-kube-api-access-9hbc8\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.678677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.678658    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-certs\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.678792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.678730    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-token\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.678792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.678757    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-socket\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.678792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.678779    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-envoy\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.678935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.678881    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-credential-socket\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.678986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.678952    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-podinfo\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.739791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.739484    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:09.780375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780347    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istiod-ca-cert\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.780572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780385    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-data\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.780572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780445    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbc8\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-kube-api-access-9hbc8\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.780572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780552    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-certs\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.780795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780600    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-token\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.780795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780747    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-socket\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.781004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780815    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-envoy\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.781004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780878    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-credential-socket\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.781004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780890    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-data\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.781004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.780975    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-podinfo\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.781198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.781071    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-certs\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.781198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.781106    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-socket\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.781198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.781135    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istiod-ca-cert\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.781355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.781267    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-credential-socket\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.783290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.783262    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-envoy\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.783670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.783654    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-podinfo\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.792586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.792563    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-token\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.793913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.793873    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbc8\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-kube-api-access-9hbc8\") pod \"e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") " pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:09.909096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.908995    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:09.973558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:09.973011    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:10.350514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:10.350431    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7" event={"ID":"eb39302f-43cf-420b-a305-73813e446ec0","Type":"ContainerStarted","Data":"8adced0753e760b392cf400eb0f37499535e5100b5290c84ce101df8605b4b6d"}
Oct 30 19:28:10.350514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:10.350495    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7" event={"ID":"eb39302f-43cf-420b-a305-73813e446ec0","Type":"ContainerStarted","Data":"d9ea9a2dc99188a56881b2e67a3ad4a522baf5e221b713012e69129b23a4d2ab"}
Oct 30 19:28:10.375357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:10.375295    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7" podStartSLOduration=2.375277641 podStartE2EDuration="2.375277641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:10.367941721 +0000 UTC m=+1325.270954759" watchObservedRunningTime="2026-10-30 19:28:10.375277641 +0000 UTC m=+1325.278290679"
Oct 30 19:28:10.381847 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:10.381818    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fa6f2a_2cef_4d7d_8128_58ee4b8107d9.slice/crio-4ed97341a30657bd7d2d1ca73c9e4ddd393001430a5914fd45f7c73194cdf445 WatchSource:0}: Error finding container 4ed97341a30657bd7d2d1ca73c9e4ddd393001430a5914fd45f7c73194cdf445: Status 404 returned error can't find the container with id 4ed97341a30657bd7d2d1ca73c9e4ddd393001430a5914fd45f7c73194cdf445
Oct 30 19:28:10.382645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:10.382625    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"]
Oct 30 19:28:10.385229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:10.385195    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:10.385355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:10.385272    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:10.385355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:10.385318    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:10.574210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:10.571567    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:10.906467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:10.906413    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:11.118773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.117738    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"]
Oct 30 19:28:11.156714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.156221    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:11.191718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.191002    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b518a7-gvx82"]
Oct 30 19:28:11.305712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.304818    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="737c41c7-66b4-4bbb-a8bd-0bdb2b562bde" path="/var/lib/kubelet/pods/737c41c7-66b4-4bbb-a8bd-0bdb2b562bde/volumes"
Oct 30 19:28:11.366527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.366493    4925 generic.go:356] "Generic (PLEG): container finished" podID="eb39302f-43cf-420b-a305-73813e446ec0" containerID="8adced0753e760b392cf400eb0f37499535e5100b5290c84ce101df8605b4b6d" exitCode=0
Oct 30 19:28:11.367019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.366710    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7" event={"ID":"eb39302f-43cf-420b-a305-73813e446ec0","Type":"ContainerDied","Data":"8adced0753e760b392cf400eb0f37499535e5100b5290c84ce101df8605b4b6d"}
Oct 30 19:28:11.378276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.378187    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" event={"ID":"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9","Type":"ContainerStarted","Data":"04ad9f219c6fbe81c0c26e0679841c4f2144de70c979e0ccadc6fc559a6d6234"}
Oct 30 19:28:11.378276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.378252    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" event={"ID":"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9","Type":"ContainerStarted","Data":"4ed97341a30657bd7d2d1ca73c9e4ddd393001430a5914fd45f7c73194cdf445"}
Oct 30 19:28:11.412518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.412447    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podStartSLOduration=2.412428706 podStartE2EDuration="2.412428706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:11.41111829 +0000 UTC m=+1326.314131329" watchObservedRunningTime="2026-10-30 19:28:11.412428706 +0000 UTC m=+1326.315441746"
Oct 30 19:28:11.553626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.553589    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:11.976167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.976135    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:11.982675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:11.982654    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:12.386350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.386260    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:12.387434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.387417    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:28:12.522533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.522499    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:12.656908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.656886    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:12.760220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.760187    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb39302f-43cf-420b-a305-73813e446ec0-kube-api-access-fzpgz\" (UniqueName: \"kubernetes.io/projected/eb39302f-43cf-420b-a305-73813e446ec0-kube-api-access-fzpgz\") pod \"eb39302f-43cf-420b-a305-73813e446ec0\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") "
Oct 30 19:28:12.760406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.760232    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-xks-ca\") pod \"eb39302f-43cf-420b-a305-73813e446ec0\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") "
Oct 30 19:28:12.760564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.760547    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-openshift-ca\") pod \"eb39302f-43cf-420b-a305-73813e446ec0\" (UID: \"eb39302f-43cf-420b-a305-73813e446ec0\") "
Oct 30 19:28:12.760888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.760866    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-openshift-ca" pod "eb39302f-43cf-420b-a305-73813e446ec0" (UID: "eb39302f-43cf-420b-a305-73813e446ec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:12.761262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.761246    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:12.762572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.762553    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-xks-ca" pod "eb39302f-43cf-420b-a305-73813e446ec0" (UID: "eb39302f-43cf-420b-a305-73813e446ec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:12.762663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.762626    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb39302f-43cf-420b-a305-73813e446ec0-kube-api-access-fzpgz" pod "eb39302f-43cf-420b-a305-73813e446ec0" (UID: "eb39302f-43cf-420b-a305-73813e446ec0"). InnerVolumeSpecName "kube-api-access-fzpgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:12.862566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.862541    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzpgz\" (UniqueName: \"kubernetes.io/projected/eb39302f-43cf-420b-a305-73813e446ec0-kube-api-access-fzpgz\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:12.862566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:12.862566    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb39302f-43cf-420b-a305-73813e446ec0-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:13.059573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:13.059468    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"]
Oct 30 19:28:13.062408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:13.062377    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:13.064949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:13.064922    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de902364-zgwbm"]
Oct 30 19:28:13.264611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:13.264572    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af99c774-060e-4e69-913b-4b3657025202" path="/var/lib/kubelet/pods/af99c774-060e-4e69-913b-4b3657025202/volumes"
Oct 30 19:28:13.347779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:13.341227    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:13.399497 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:13.399471    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"
Oct 30 19:28:13.399997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:13.399504    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7" event={"ID":"eb39302f-43cf-420b-a305-73813e446ec0","Type":"ContainerDied","Data":"d9ea9a2dc99188a56881b2e67a3ad4a522baf5e221b713012e69129b23a4d2ab"}
Oct 30 19:28:13.399997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:13.399563    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9ea9a2dc99188a56881b2e67a3ad4a522baf5e221b713012e69129b23a4d2ab"
Oct 30 19:28:13.875853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:13.875820    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:14.071987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:14.071952    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:14.354577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:14.354546    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:14.563870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:14.563843    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:14.915716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:14.914399    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:15.479713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.479633    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:15.538088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.538062    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"]
Oct 30 19:28:15.538830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.538814    4925 cpu_manager.go:414] "containerMap: removing container" podUID="eb39302f-43cf-420b-a305-73813e446ec0" containerName="revoke-keys"
Oct 30 19:28:15.538892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.538830    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb39302f-43cf-420b-a305-73813e446ec0" containerName="revoke-keys"
Oct 30 19:28:15.538956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.538928    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb39302f-43cf-420b-a305-73813e446ec0" containerName="revoke-keys"
Oct 30 19:28:15.543785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.543766    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:15.552623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.552600    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8ceda0\""
Oct 30 19:28:15.553504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.552786    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"]
Oct 30 19:28:15.647727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.635383    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8ceda0-dockercfg-8kwt4\""
Oct 30 19:28:15.688669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.666029    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049d90d-2ea9-4e0a-8cca-b3261600f27e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659\" (UID: \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:15.688669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.666233    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgcfh\" (UniqueName: \"kubernetes.io/projected/e049d90d-2ea9-4e0a-8cca-b3261600f27e-kube-api-access-pgcfh\") pod \"payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659\" (UID: \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:15.689628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.689594    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:15.852439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.852340    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049d90d-2ea9-4e0a-8cca-b3261600f27e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659\" (UID: \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:15.881716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.852876    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgcfh\" (UniqueName: \"kubernetes.io/projected/e049d90d-2ea9-4e0a-8cca-b3261600f27e-kube-api-access-pgcfh\") pod \"payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659\" (UID: \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:15.931122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.854737    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049d90d-2ea9-4e0a-8cca-b3261600f27e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659\" (UID: \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:15.956719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.952529    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:15.956719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:15.955269    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgcfh\" (UniqueName: \"kubernetes.io/projected/e049d90d-2ea9-4e0a-8cca-b3261600f27e-kube-api-access-pgcfh\") pod \"payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659\" (UID: \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:16.181815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:16.181783    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:16.494086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:16.493946    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"]
Oct 30 19:28:16.495102 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:16.495073    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode049d90d_2ea9_4e0a_8cca_b3261600f27e.slice/crio-b66b802802c01880c7979c0f50fda73a3286ac9fcf5b5a4a9ad04ddfa7ce056e WatchSource:0}: Error finding container b66b802802c01880c7979c0f50fda73a3286ac9fcf5b5a4a9ad04ddfa7ce056e: Status 404 returned error can't find the container with id b66b802802c01880c7979c0f50fda73a3286ac9fcf5b5a4a9ad04ddfa7ce056e
Oct 30 19:28:16.668541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:16.668510    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:17.463714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:17.459171    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:17.463714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:17.462321    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659" event={"ID":"e049d90d-2ea9-4e0a-8cca-b3261600f27e","Type":"ContainerStarted","Data":"b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c"}
Oct 30 19:28:17.463714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:17.462377    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659" event={"ID":"e049d90d-2ea9-4e0a-8cca-b3261600f27e","Type":"ContainerStarted","Data":"b66b802802c01880c7979c0f50fda73a3286ac9fcf5b5a4a9ad04ddfa7ce056e"}
Oct 30 19:28:17.463714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:17.463292    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:17.486632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:17.486576    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659" podStartSLOduration=2.486561528 podStartE2EDuration="2.486561528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:17.484064941 +0000 UTC m=+1332.387077970" watchObservedRunningTime="2026-10-30 19:28:17.486561528 +0000 UTC m=+1332.389574566"
Oct 30 19:28:18.033162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:18.033102    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"]
Oct 30 19:28:18.040113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:18.040082    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a95ac3a-kpf7w"]
Oct 30 19:28:18.265825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:18.265794    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:18.724490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:18.724459    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"]
Oct 30 19:28:18.873329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:18.873299    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:19.050769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:19.045259    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:19.271147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:19.271111    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7" path="/var/lib/kubelet/pods/68f7bc46-c6e2-4011-b6dd-4cd0f4d4c8f7/volumes"
Oct 30 19:28:19.871208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:19.870934    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:20.133609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:20.133533    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:20.347617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:20.347579    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:20.536444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:20.536414    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:20.746481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:20.746443    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:20.945310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:20.945268    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:21.497440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:21.497407    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:21.530400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:21.530372    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"]
Oct 30 19:28:21.537439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:21.537419    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d360546-dwfkw"]
Oct 30 19:28:21.702346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:21.702309    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:21.992987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:21.992959    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:22.045574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:22.045194    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"]
Oct 30 19:28:22.050456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:22.050419    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-451fba18-hnsnn"]
Oct 30 19:28:22.131261 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:28:22.131224    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:28:22.224713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:22.219071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:22.983781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:22.983745    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:23.263995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:23.263923    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3336a061-d04a-4b02-914d-42941df3d8a1" path="/var/lib/kubelet/pods/3336a061-d04a-4b02-914d-42941df3d8a1/volumes"
Oct 30 19:28:23.264392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:23.264378    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bdfd5e-681a-4d20-873d-9b7f8cee77e5" path="/var/lib/kubelet/pods/53bdfd5e-681a-4d20-873d-9b7f8cee77e5/volumes"
Oct 30 19:28:23.726454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:23.726413    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs" podUID="24993ca9-1cac-4bff-8c7f-813c98303bf6" containerName="payload-processing" containerID="cri-o://322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa" gracePeriod=25
Oct 30 19:28:23.750001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:23.749969    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:24.097455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.097430    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:28:24.153006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.152979    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24993ca9-1cac-4bff-8c7f-813c98303bf6-kube-api-access-646ht\" (UniqueName: \"kubernetes.io/projected/24993ca9-1cac-4bff-8c7f-813c98303bf6-kube-api-access-646ht\") pod \"24993ca9-1cac-4bff-8c7f-813c98303bf6\" (UID: \"24993ca9-1cac-4bff-8c7f-813c98303bf6\") "
Oct 30 19:28:24.153213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.153087    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24993ca9-1cac-4bff-8c7f-813c98303bf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24993ca9-1cac-4bff-8c7f-813c98303bf6-plugins-config-volume\") pod \"24993ca9-1cac-4bff-8c7f-813c98303bf6\" (UID: \"24993ca9-1cac-4bff-8c7f-813c98303bf6\") "
Oct 30 19:28:24.153463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.153442    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24993ca9-1cac-4bff-8c7f-813c98303bf6-plugins-config-volume" pod "24993ca9-1cac-4bff-8c7f-813c98303bf6" (UID: "24993ca9-1cac-4bff-8c7f-813c98303bf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:24.153908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.153882    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24993ca9-1cac-4bff-8c7f-813c98303bf6-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:24.155498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.155476    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24993ca9-1cac-4bff-8c7f-813c98303bf6-kube-api-access-646ht" pod "24993ca9-1cac-4bff-8c7f-813c98303bf6" (UID: "24993ca9-1cac-4bff-8c7f-813c98303bf6"). InnerVolumeSpecName "kube-api-access-646ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:24.256115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.256076    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646ht\" (UniqueName: \"kubernetes.io/projected/24993ca9-1cac-4bff-8c7f-813c98303bf6-kube-api-access-646ht\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:24.535856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.535812    4925 generic.go:356] "Generic (PLEG): container finished" podID="24993ca9-1cac-4bff-8c7f-813c98303bf6" containerID="322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa" exitCode=0
Oct 30 19:28:24.536030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.535868    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"
Oct 30 19:28:24.536030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.535887    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs" event={"ID":"24993ca9-1cac-4bff-8c7f-813c98303bf6","Type":"ContainerDied","Data":"322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa"}
Oct 30 19:28:24.536030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.535939    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs" event={"ID":"24993ca9-1cac-4bff-8c7f-813c98303bf6","Type":"ContainerDied","Data":"04399bc56dba4ce2f9dfbbc01645f26350eb8fa3be9426253226a479ba02e9c6"}
Oct 30 19:28:24.536030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.535967    4925 scope.go:122] "RemoveContainer" containerID="322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa"
Oct 30 19:28:24.548284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.548265    4925 scope.go:122] "RemoveContainer" containerID="322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa"
Oct 30 19:28:24.548543 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:28:24.548523    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa\": container with ID starting with 322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa not found: ID does not exist" containerID="322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa"
Oct 30 19:28:24.548627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.548551    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa"} err="failed to get container status \"322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa\": rpc error: code = NotFound desc = could not find container \"322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa\": container with ID starting with 322686d3825ad17faffe6d506bbf78af7dc4b666fba4ebbb27d1ddf05c48fafa not found: ID does not exist"
Oct 30 19:28:24.561866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.561840    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"]
Oct 30 19:28:24.566664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.566643    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a718a-6f7445cb4d-h2hxs"]
Oct 30 19:28:24.642524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:24.642492    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:25.264736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:25.264670    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24993ca9-1cac-4bff-8c7f-813c98303bf6" path="/var/lib/kubelet/pods/24993ca9-1cac-4bff-8c7f-813c98303bf6/volumes"
Oct 30 19:28:25.473728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:25.473698    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:26.172453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:26.172377    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:26.961633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:26.961590    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:27.663876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:27.663845    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:28.360854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.360827    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:28.605639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.605612    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"]
Oct 30 19:28:28.606356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.606342    4925 cpu_manager.go:414] "containerMap: removing container" podUID="24993ca9-1cac-4bff-8c7f-813c98303bf6" containerName="payload-processing"
Oct 30 19:28:28.606409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.606357    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24993ca9-1cac-4bff-8c7f-813c98303bf6" containerName="payload-processing"
Oct 30 19:28:28.606498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.606488    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="24993ca9-1cac-4bff-8c7f-813c98303bf6" containerName="payload-processing"
Oct 30 19:28:28.611611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.611569    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.662219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.662194    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:28.662359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.662291    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:28:28.664902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.664880    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"]
Oct 30 19:28:28.668404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.668379    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:28.675988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.675960    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpqw\" (UniqueName: \"kubernetes.io/projected/f02488c1-623d-4da3-86d3-9141f35072b6-kube-api-access-zlpqw\") pod \"maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.676236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.676214    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.676311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.676250    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.777827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.777796    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.777827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.777832    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.778060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.778002    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpqw\" (UniqueName: \"kubernetes.io/projected/f02488c1-623d-4da3-86d3-9141f35072b6-kube-api-access-zlpqw\") pod \"maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.778436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.778415    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.780364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.780339    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.791407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.791385    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpqw\" (UniqueName: \"kubernetes.io/projected/f02488c1-623d-4da3-86d3-9141f35072b6-kube-api-access-zlpqw\") pod \"maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.926133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.926102    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:28.970780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:28.970752    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:29.243776 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:29.243746    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02488c1_623d_4da3_86d3_9141f35072b6.slice/crio-2f05d31a8237e441ded1d3161cb68f9ef6d34096daffb3563a471540707f857d WatchSource:0}: Error finding container 2f05d31a8237e441ded1d3161cb68f9ef6d34096daffb3563a471540707f857d: Status 404 returned error can't find the container with id 2f05d31a8237e441ded1d3161cb68f9ef6d34096daffb3563a471540707f857d
Oct 30 19:28:29.244052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.244031    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"]
Oct 30 19:28:29.330388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.330353    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"]
Oct 30 19:28:29.366578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.357095    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.366578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.358529    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"]
Oct 30 19:28:29.415071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.415037    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-33b4d7-openshift-default-dockercfg-2x2w9\""
Oct 30 19:28:29.481600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.481566    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:29.497775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.493877    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-envoy\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.497775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.494026    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-certs\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.497775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.494086    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febce7a9-ce87-4830-9d20-9ace649530e2-istiod-ca-cert\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.497775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.494141    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.497775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.494275    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-data\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.497775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.494312    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhf2p\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-kube-api-access-bhf2p\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.497775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.494369    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-credential-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.497775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.494491    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-istio-token\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.497775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.494520    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febce7a9-ce87-4830-9d20-9ace649530e2-istio-podinfo\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.530819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.525960    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.628861    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-istio-token\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.628913    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febce7a9-ce87-4830-9d20-9ace649530e2-istio-podinfo\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.629046    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-envoy\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.629168    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-certs\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.629225    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febce7a9-ce87-4830-9d20-9ace649530e2-istiod-ca-cert\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.629281    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.629454    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-data\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.629484    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhf2p\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-kube-api-access-bhf2p\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.629545    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-credential-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.630065    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-credential-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.646290    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febce7a9-ce87-4830-9d20-9ace649530e2-istiod-ca-cert\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.647506    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-socket\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.647785    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-data\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.662975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.648070    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-certs\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.666624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.664763    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-envoy\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.669939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.669867    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-istio-token\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.672605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.672537    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhf2p\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-kube-api-access-bhf2p\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.673365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.673300    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febce7a9-ce87-4830-9d20-9ace649530e2-istio-podinfo\") pod \"e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") " pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.705049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.680068    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:29.784707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.768892    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:29.919378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.911481    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849" event={"ID":"f02488c1-623d-4da3-86d3-9141f35072b6","Type":"ContainerStarted","Data":"2f05d31a8237e441ded1d3161cb68f9ef6d34096daffb3563a471540707f857d"}
Oct 30 19:28:29.949081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:29.948960    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:30.115392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.114528    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:30.363734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.363667    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"]
Oct 30 19:28:30.365494 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:30.365421    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebce7a9_ce87_4830_9d20_9ace649530e2.slice/crio-835f69b807acced267c68cf1a0286133e5eacd822a195e4cc2cbff5c8c50ed24 WatchSource:0}: Error finding container 835f69b807acced267c68cf1a0286133e5eacd822a195e4cc2cbff5c8c50ed24: Status 404 returned error can't find the container with id 835f69b807acced267c68cf1a0286133e5eacd822a195e4cc2cbff5c8c50ed24
Oct 30 19:28:30.369641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.369602    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:30.369964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.369752    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:30.369964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.369801    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:30.927519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.927482    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" event={"ID":"febce7a9-ce87-4830-9d20-9ace649530e2","Type":"ContainerStarted","Data":"6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b"}
Oct 30 19:28:30.927755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.927548    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" event={"ID":"febce7a9-ce87-4830-9d20-9ace649530e2","Type":"ContainerStarted","Data":"835f69b807acced267c68cf1a0286133e5eacd822a195e4cc2cbff5c8c50ed24"}
Oct 30 19:28:30.929819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.929791    4925 generic.go:356] "Generic (PLEG): container finished" podID="f02488c1-623d-4da3-86d3-9141f35072b6" containerID="51a33b780f043bacdca2857f735ef31f37ff3e54e90f0668e1bc558986a34a0d" exitCode=0
Oct 30 19:28:30.929953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.929832    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849" event={"ID":"f02488c1-623d-4da3-86d3-9141f35072b6","Type":"ContainerDied","Data":"51a33b780f043bacdca2857f735ef31f37ff3e54e90f0668e1bc558986a34a0d"}
Oct 30 19:28:30.955781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.955733    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" podStartSLOduration=1.955721322 podStartE2EDuration="1.955721322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:30.953732812 +0000 UTC m=+1345.856745849" watchObservedRunningTime="2026-10-30 19:28:30.955721322 +0000 UTC m=+1345.858734359"
Oct 30 19:28:30.993118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:30.993020    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:31.040254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:31.040224    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"]
Oct 30 19:28:31.044512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:31.044485    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce19e59-kc9f5"]
Oct 30 19:28:31.181975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:31.181941    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:31.264991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:31.264958    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e6cbf8-5117-45b2-9527-28a77a579d00" path="/var/lib/kubelet/pods/82e6cbf8-5117-45b2-9527-28a77a579d00/volumes"
Oct 30 19:28:31.447762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:31.443073    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:31.687059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:31.687031    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:31.733768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:31.726055    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:31.938230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:31.938199    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:31.939285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:31.939265    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:28:32.049642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.049609    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:32.237893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.237870    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:32.352831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.352754    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-openshift-ca\") pod \"f02488c1-623d-4da3-86d3-9141f35072b6\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") "
Oct 30 19:28:32.352976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.352920    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02488c1-623d-4da3-86d3-9141f35072b6-kube-api-access-zlpqw\" (UniqueName: \"kubernetes.io/projected/f02488c1-623d-4da3-86d3-9141f35072b6-kube-api-access-zlpqw\") pod \"f02488c1-623d-4da3-86d3-9141f35072b6\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") "
Oct 30 19:28:32.353118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.353102    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-xks-ca\") pod \"f02488c1-623d-4da3-86d3-9141f35072b6\" (UID: \"f02488c1-623d-4da3-86d3-9141f35072b6\") "
Oct 30 19:28:32.353156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.353136    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-openshift-ca" pod "f02488c1-623d-4da3-86d3-9141f35072b6" (UID: "f02488c1-623d-4da3-86d3-9141f35072b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:32.354078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.354055    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:32.355353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.355330    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02488c1-623d-4da3-86d3-9141f35072b6-kube-api-access-zlpqw" pod "f02488c1-623d-4da3-86d3-9141f35072b6" (UID: "f02488c1-623d-4da3-86d3-9141f35072b6"). InnerVolumeSpecName "kube-api-access-zlpqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:32.355457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.355394    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-xks-ca" pod "f02488c1-623d-4da3-86d3-9141f35072b6" (UID: "f02488c1-623d-4da3-86d3-9141f35072b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:32.455576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.455545    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlpqw\" (UniqueName: \"kubernetes.io/projected/f02488c1-623d-4da3-86d3-9141f35072b6-kube-api-access-zlpqw\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:32.455576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.455573    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f02488c1-623d-4da3-86d3-9141f35072b6-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:32.947621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.947589    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"
Oct 30 19:28:32.947794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.947618    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849" event={"ID":"f02488c1-623d-4da3-86d3-9141f35072b6","Type":"ContainerDied","Data":"2f05d31a8237e441ded1d3161cb68f9ef6d34096daffb3563a471540707f857d"}
Oct 30 19:28:32.947794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:32.947701    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f05d31a8237e441ded1d3161cb68f9ef6d34096daffb3563a471540707f857d"
Oct 30 19:28:33.081595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:33.081554    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:33.455590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:33.455562    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:33.630994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:33.630960    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:33.896533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:33.896500    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:34.076746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.076713    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:34.248147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.248053    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"]
Oct 30 19:28:34.249258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.249219    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f02488c1-623d-4da3-86d3-9141f35072b6" containerName="revoke-keys"
Oct 30 19:28:34.249258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.249244    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02488c1-623d-4da3-86d3-9141f35072b6" containerName="revoke-keys"
Oct 30 19:28:34.249472 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.249463    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02488c1-623d-4da3-86d3-9141f35072b6" containerName="revoke-keys"
Oct 30 19:28:34.254356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.254333    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.264779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.264742    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-891d80-openshift-default-dockercfg-lk8xn\""
Oct 30 19:28:34.269360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.269334    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"]
Oct 30 19:28:34.284005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.283971    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istiod-ca-cert\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.284137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.284042    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-envoy\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.284252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.284233    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-token\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.284356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.284339    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-podinfo\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.284434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.284395    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.284499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.284463    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwksh\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-kube-api-access-cwksh\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.284558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.284526    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-data\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.284616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.284575    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-certs\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.284616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.284610    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-credential-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.348136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.348111    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:34.385766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.385736    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-token\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.385892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.385834    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-podinfo\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.385892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.385873    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.385904    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwksh\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-kube-api-access-cwksh\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.385947    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-data\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.385988    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-certs\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.386019    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-credential-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.386102    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istiod-ca-cert\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.386155    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-envoy\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.386348    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.386505    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-data\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.386520    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-credential-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.386593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.386563    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-certs\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.387162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.387084    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istiod-ca-cert\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.388664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.388645    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-envoy\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.389048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.389029    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-podinfo\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.400625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.400598    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-token\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.415933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.415865    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwksh\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-kube-api-access-cwksh\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-z4592\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:34.552268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.552123    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:34.580935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:34.579983    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:35.062526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:35.062443    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"]
Oct 30 19:28:35.066102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:35.065934    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:35.069734 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:35.069678    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be73c9c_3cd1_40a0_8ad3_66c731ae7ecd.slice/crio-5f42dc163235e3ef91d46605eac4e23a43c6f5dd3eabe599f4ebaf1fa17d816c WatchSource:0}: Error finding container 5f42dc163235e3ef91d46605eac4e23a43c6f5dd3eabe599f4ebaf1fa17d816c: Status 404 returned error can't find the container with id 5f42dc163235e3ef91d46605eac4e23a43c6f5dd3eabe599f4ebaf1fa17d816c
Oct 30 19:28:35.072625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:35.072595    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:35.072721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:35.072675    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:35.072775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:35.072759    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:28:35.276951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:35.268859    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:35.976536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:35.976502    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" event={"ID":"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd","Type":"ContainerStarted","Data":"a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d"}
Oct 30 19:28:35.977246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:35.977216    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" event={"ID":"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd","Type":"ContainerStarted","Data":"5f42dc163235e3ef91d46605eac4e23a43c6f5dd3eabe599f4ebaf1fa17d816c"}
Oct 30 19:28:36.000870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:36.000820    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podStartSLOduration=2.000803201 podStartE2EDuration="2.000803201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:35.997199556 +0000 UTC m=+1350.900212593" watchObservedRunningTime="2026-10-30 19:28:36.000803201 +0000 UTC m=+1350.903816233"
Oct 30 19:28:36.181716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:36.180671    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:36.581697 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:36.581660    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:36.587873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:36.587849    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:36.988135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:36.988109    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:36.989010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:36.988989    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:28:37.040406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:37.040373    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"]
Oct 30 19:28:37.045828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:37.045792    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87dc7c05-nw46g"]
Oct 30 19:28:37.261094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:37.260902    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:37.288945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:37.288912    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75ddf1c-5514-46f3-83b7-d0c8cf941aa2" path="/var/lib/kubelet/pods/b75ddf1c-5514-46f3-83b7-d0c8cf941aa2/volumes"
Oct 30 19:28:38.031100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:38.031072    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"]
Oct 30 19:28:38.038171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:38.038144    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d925576b-xzxs9"]
Oct 30 19:28:38.315263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:38.315172    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:38.529442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:38.529408    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:38.705942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:38.705902    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:38.710051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:38.710018    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"]
Oct 30 19:28:38.718914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:38.718891    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"]
Oct 30 19:28:38.882702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:38.882647    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:39.280307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:39.280273    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d8959d-aca3-4c20-8558-3335f9b44729" path="/var/lib/kubelet/pods/f4d8959d-aca3-4c20-8558-3335f9b44729/volumes"
Oct 30 19:28:39.958219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:39.956231    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:40.128947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:40.128917    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:40.302768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:40.302634    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:41.000416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:41.000382    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:41.172787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:41.172753    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:41.658502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:41.658394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:42.666889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:42.663679    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:43.361236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:43.361168    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:43.711314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:43.711266    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs" podUID="dd4240a3-839f-479d-83c9-526adaef0013" containerName="payload-processing" containerID="cri-o://0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde" gracePeriod=25
Oct 30 19:28:43.719782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:43.719748    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz" podUID="25348f9a-6e93-4fb7-b127-51b6910e514b" containerName="payload-pre-processing" containerID="cri-o://2aaf9ff8118e511e11efa78c953050da72ceb4eb489e5d3fc5bb4d8ab15fd8de" gracePeriod=25
Oct 30 19:28:44.005537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.005493    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:28:44.114245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.114196    4925 generic.go:356] "Generic (PLEG): container finished" podID="dd4240a3-839f-479d-83c9-526adaef0013" containerID="0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde" exitCode=0
Oct 30 19:28:44.114432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.114414    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs" event={"ID":"dd4240a3-839f-479d-83c9-526adaef0013","Type":"ContainerDied","Data":"0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde"}
Oct 30 19:28:44.114486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.114469    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs" event={"ID":"dd4240a3-839f-479d-83c9-526adaef0013","Type":"ContainerDied","Data":"7c4be612c0bcec38cd4a3da1dcaf88a341174a70ba518c83ee087ed4c554ef92"}
Oct 30 19:28:44.114522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.114507    4925 scope.go:122] "RemoveContainer" containerID="0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde"
Oct 30 19:28:44.114665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.114654    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"
Oct 30 19:28:44.118769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.118662    4925 generic.go:356] "Generic (PLEG): container finished" podID="25348f9a-6e93-4fb7-b127-51b6910e514b" containerID="2aaf9ff8118e511e11efa78c953050da72ceb4eb489e5d3fc5bb4d8ab15fd8de" exitCode=0
Oct 30 19:28:44.119203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.118960    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz" event={"ID":"25348f9a-6e93-4fb7-b127-51b6910e514b","Type":"ContainerDied","Data":"2aaf9ff8118e511e11efa78c953050da72ceb4eb489e5d3fc5bb4d8ab15fd8de"}
Oct 30 19:28:44.130790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.126235    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:44.146267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.144478    4925 scope.go:122] "RemoveContainer" containerID="0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde"
Oct 30 19:28:44.146870 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:28:44.146819    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde\": container with ID starting with 0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde not found: ID does not exist" containerID="0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde"
Oct 30 19:28:44.147048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.147020    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde"} err="failed to get container status \"0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde\": rpc error: code = NotFound desc = could not find container \"0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde\": container with ID starting with 0e5e0a0583f9c21e1ee8704b58e7782c73e51cd6a14467f531bd9acca7019dde not found: ID does not exist"
Oct 30 19:28:44.158904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.158885    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd4240a3-839f-479d-83c9-526adaef0013-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd4240a3-839f-479d-83c9-526adaef0013-plugins-config-volume\") pod \"dd4240a3-839f-479d-83c9-526adaef0013\" (UID: \"dd4240a3-839f-479d-83c9-526adaef0013\") "
Oct 30 19:28:44.159093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.159078    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4240a3-839f-479d-83c9-526adaef0013-kube-api-access-hnbmr\" (UniqueName: \"kubernetes.io/projected/dd4240a3-839f-479d-83c9-526adaef0013-kube-api-access-hnbmr\") pod \"dd4240a3-839f-479d-83c9-526adaef0013\" (UID: \"dd4240a3-839f-479d-83c9-526adaef0013\") "
Oct 30 19:28:44.159308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.159286    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4240a3-839f-479d-83c9-526adaef0013-plugins-config-volume" pod "dd4240a3-839f-479d-83c9-526adaef0013" (UID: "dd4240a3-839f-479d-83c9-526adaef0013"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:44.159756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.159736    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd4240a3-839f-479d-83c9-526adaef0013-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:44.161484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.161462    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4240a3-839f-479d-83c9-526adaef0013-kube-api-access-hnbmr" pod "dd4240a3-839f-479d-83c9-526adaef0013" (UID: "dd4240a3-839f-479d-83c9-526adaef0013"). InnerVolumeSpecName "kube-api-access-hnbmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:44.175017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.174997    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:28:44.260848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.260768    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25348f9a-6e93-4fb7-b127-51b6910e514b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25348f9a-6e93-4fb7-b127-51b6910e514b-plugins-config-volume\") pod \"25348f9a-6e93-4fb7-b127-51b6910e514b\" (UID: \"25348f9a-6e93-4fb7-b127-51b6910e514b\") "
Oct 30 19:28:44.261136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.261116    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25348f9a-6e93-4fb7-b127-51b6910e514b-kube-api-access-scw8h\" (UniqueName: \"kubernetes.io/projected/25348f9a-6e93-4fb7-b127-51b6910e514b-kube-api-access-scw8h\") pod \"25348f9a-6e93-4fb7-b127-51b6910e514b\" (UID: \"25348f9a-6e93-4fb7-b127-51b6910e514b\") "
Oct 30 19:28:44.261220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.261164    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25348f9a-6e93-4fb7-b127-51b6910e514b-plugins-config-volume" pod "25348f9a-6e93-4fb7-b127-51b6910e514b" (UID: "25348f9a-6e93-4fb7-b127-51b6910e514b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:44.261812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.261795    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnbmr\" (UniqueName: \"kubernetes.io/projected/dd4240a3-839f-479d-83c9-526adaef0013-kube-api-access-hnbmr\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:44.261877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.261814    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25348f9a-6e93-4fb7-b127-51b6910e514b-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:44.263424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.263403    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25348f9a-6e93-4fb7-b127-51b6910e514b-kube-api-access-scw8h" pod "25348f9a-6e93-4fb7-b127-51b6910e514b" (UID: "25348f9a-6e93-4fb7-b127-51b6910e514b"). InnerVolumeSpecName "kube-api-access-scw8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:44.363660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.363624    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scw8h\" (UniqueName: \"kubernetes.io/projected/25348f9a-6e93-4fb7-b127-51b6910e514b-kube-api-access-scw8h\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:44.439524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.439473    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"]
Oct 30 19:28:44.444399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.444376    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48b1a6a7-5547c65796-crcbs"]
Oct 30 19:28:44.865578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:44.865532    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:45.130353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:45.130268    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz" event={"ID":"25348f9a-6e93-4fb7-b127-51b6910e514b","Type":"ContainerDied","Data":"0ca1dd5c13fcd386e365f26fe323d195ffd03581e9861aa78b43c10c4b7dfb2c"}
Oct 30 19:28:45.130353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:45.130306    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"
Oct 30 19:28:45.130353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:45.130334    4925 scope.go:122] "RemoveContainer" containerID="2aaf9ff8118e511e11efa78c953050da72ceb4eb489e5d3fc5bb4d8ab15fd8de"
Oct 30 19:28:45.154473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:45.154446    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"]
Oct 30 19:28:45.159153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:45.159133    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-48b1a6a7-65cdd77d95-rmdbz"]
Oct 30 19:28:45.264734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:45.264705    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25348f9a-6e93-4fb7-b127-51b6910e514b" path="/var/lib/kubelet/pods/25348f9a-6e93-4fb7-b127-51b6910e514b/volumes"
Oct 30 19:28:45.265202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:45.265187    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd4240a3-839f-479d-83c9-526adaef0013" path="/var/lib/kubelet/pods/dd4240a3-839f-479d-83c9-526adaef0013/volumes"
Oct 30 19:28:45.515720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:45.514521    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:46.140263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.139990    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:46.563726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.563627    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"]
Oct 30 19:28:46.564348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.564333    4925 cpu_manager.go:414] "containerMap: removing container" podUID="25348f9a-6e93-4fb7-b127-51b6910e514b" containerName="payload-pre-processing"
Oct 30 19:28:46.564394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.564349    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25348f9a-6e93-4fb7-b127-51b6910e514b" containerName="payload-pre-processing"
Oct 30 19:28:46.564394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.564389    4925 cpu_manager.go:414] "containerMap: removing container" podUID="dd4240a3-839f-479d-83c9-526adaef0013" containerName="payload-processing"
Oct 30 19:28:46.564458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.564396    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4240a3-839f-479d-83c9-526adaef0013" containerName="payload-processing"
Oct 30 19:28:46.564529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.564520    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4240a3-839f-479d-83c9-526adaef0013" containerName="payload-processing"
Oct 30 19:28:46.564567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.564535    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="25348f9a-6e93-4fb7-b127-51b6910e514b" containerName="payload-pre-processing"
Oct 30 19:28:46.569651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.569633    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.572725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.572703    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:46.573037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.573015    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:28:46.576502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.576479    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"]
Oct 30 19:28:46.700085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.700049    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-692a3083-drsvq\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.700270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.700189    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-692a3083-drsvq\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.700270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.700225    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sxp2\" (UniqueName: \"kubernetes.io/projected/a74313a0-412d-4e3e-98f0-1bbccf4fa459-kube-api-access-8sxp2\") pod \"maas-api-revoke-sub-692a3083-drsvq\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.801538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.801508    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-692a3083-drsvq\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.801746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.801583    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-692a3083-drsvq\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.801746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.801610    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sxp2\" (UniqueName: \"kubernetes.io/projected/a74313a0-412d-4e3e-98f0-1bbccf4fa459-kube-api-access-8sxp2\") pod \"maas-api-revoke-sub-692a3083-drsvq\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.802399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.802375    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-692a3083-drsvq\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.804314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.804288    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-692a3083-drsvq\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.813783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.813720    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sxp2\" (UniqueName: \"kubernetes.io/projected/a74313a0-412d-4e3e-98f0-1bbccf4fa459-kube-api-access-8sxp2\") pod \"maas-api-revoke-sub-692a3083-drsvq\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.888198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.888171    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:46.926421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:46.926376    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:47.254231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:47.254203    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"]
Oct 30 19:28:47.254761 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:47.254727    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74313a0_412d_4e3e_98f0_1bbccf4fa459.slice/crio-b87c177aacd4c4f948d8ec8a761efcfc93f093ed95844a6c8257b3fb2043d335 WatchSource:0}: Error finding container b87c177aacd4c4f948d8ec8a761efcfc93f093ed95844a6c8257b3fb2043d335: Status 404 returned error can't find the container with id b87c177aacd4c4f948d8ec8a761efcfc93f093ed95844a6c8257b3fb2043d335
Oct 30 19:28:47.889295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:47.889263    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:48.185398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:48.185366    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq" event={"ID":"a74313a0-412d-4e3e-98f0-1bbccf4fa459","Type":"ContainerStarted","Data":"3e9f6f88db228275820f17c8ba0ceaed00a22ce50e00ac67b9a1b9ee0046bd1e"}
Oct 30 19:28:48.185591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:48.185415    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq" event={"ID":"a74313a0-412d-4e3e-98f0-1bbccf4fa459","Type":"ContainerStarted","Data":"b87c177aacd4c4f948d8ec8a761efcfc93f093ed95844a6c8257b3fb2043d335"}
Oct 30 19:28:48.203940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:48.203896    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq" podStartSLOduration=2.203881456 podStartE2EDuration="2.203881456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:48.201110215 +0000 UTC m=+1363.104123252" watchObservedRunningTime="2026-10-30 19:28:48.203881456 +0000 UTC m=+1363.106894517"
Oct 30 19:28:48.876712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:48.876052    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:49.205218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.205175    4925 generic.go:356] "Generic (PLEG): container finished" podID="a74313a0-412d-4e3e-98f0-1bbccf4fa459" containerID="3e9f6f88db228275820f17c8ba0ceaed00a22ce50e00ac67b9a1b9ee0046bd1e" exitCode=0
Oct 30 19:28:49.205483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.205464    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq" event={"ID":"a74313a0-412d-4e3e-98f0-1bbccf4fa459","Type":"ContainerDied","Data":"3e9f6f88db228275820f17c8ba0ceaed00a22ce50e00ac67b9a1b9ee0046bd1e"}
Oct 30 19:28:49.434145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.434099    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:49.873669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.873620    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:49.881736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.880805    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"]
Oct 30 19:28:49.898554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.887642    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"]
Oct 30 19:28:49.898554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.887794    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:49.898554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.893095    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-33b4d7-serving-cert\""
Oct 30 19:28:49.960243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.960109    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b500b6-9c01-4197-899a-3e72217112a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:49.961661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.960811    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djn9h\" (UniqueName: \"kubernetes.io/projected/c3b500b6-9c01-4197-899a-3e72217112a4-kube-api-access-djn9h\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:49.962770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.962748    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-metrics-certs\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:49.963049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:49.963030    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.069714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.065095    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b500b6-9c01-4197-899a-3e72217112a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.069714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.065390    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djn9h\" (UniqueName: \"kubernetes.io/projected/c3b500b6-9c01-4197-899a-3e72217112a4-kube-api-access-djn9h\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.069714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.065428    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-metrics-certs\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.069714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.065535    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.081359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.081304    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b500b6-9c01-4197-899a-3e72217112a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.081846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.081822    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:50.137890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.136555    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-metrics-certs\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.137890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.137144    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.144880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.142550    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djn9h\" (UniqueName: \"kubernetes.io/projected/c3b500b6-9c01-4197-899a-3e72217112a4-kube-api-access-djn9h\") pod \"maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.226360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.226320    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:50.238401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.238365    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:50.759818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.756886    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"]
Oct 30 19:28:50.759818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.757171    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7" podUID="eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" containerName="maas-api" containerID="cri-o://3ef3ad607dba6e21b248c93ba67c0f7041e4b43f5ac1a88e56e94505ff7915d5" gracePeriod=30
Oct 30 19:28:50.892234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:50.892211    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:51.162713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.156016    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:51.162713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.158168    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:51.220874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.219376    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-xks-ca\") pod \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") "
Oct 30 19:28:51.220874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.219788    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-openshift-ca\") pod \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") "
Oct 30 19:28:51.220874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.219893    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74313a0-412d-4e3e-98f0-1bbccf4fa459-kube-api-access-8sxp2\" (UniqueName: \"kubernetes.io/projected/a74313a0-412d-4e3e-98f0-1bbccf4fa459-kube-api-access-8sxp2\") pod \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\" (UID: \"a74313a0-412d-4e3e-98f0-1bbccf4fa459\") "
Oct 30 19:28:51.222402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.222377    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"]
Oct 30 19:28:51.339014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.232388    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-openshift-ca" pod "a74313a0-412d-4e3e-98f0-1bbccf4fa459" (UID: "a74313a0-412d-4e3e-98f0-1bbccf4fa459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:51.375789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.363107    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:51.465720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.447061    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:51.545708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.539378    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-xks-ca" pod "a74313a0-412d-4e3e-98f0-1bbccf4fa459" (UID: "a74313a0-412d-4e3e-98f0-1bbccf4fa459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:51.587128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.587079    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74313a0-412d-4e3e-98f0-1bbccf4fa459-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:51.606676 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:51.606638    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b500b6_9c01_4197_899a_3e72217112a4.slice/crio-e4eb4bc23940046d4442530330beb79ed27f8d44c6a8a7f1a3364ad460ae444e WatchSource:0}: Error finding container e4eb4bc23940046d4442530330beb79ed27f8d44c6a8a7f1a3364ad460ae444e: Status 404 returned error can't find the container with id e4eb4bc23940046d4442530330beb79ed27f8d44c6a8a7f1a3364ad460ae444e
Oct 30 19:28:51.640309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.638647    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74313a0-412d-4e3e-98f0-1bbccf4fa459-kube-api-access-8sxp2" pod "a74313a0-412d-4e3e-98f0-1bbccf4fa459" (UID: "a74313a0-412d-4e3e-98f0-1bbccf4fa459"). InnerVolumeSpecName "kube-api-access-8sxp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:51.756174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.755992    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:51.775369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.775326    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9a718a-zl9w7"]
Oct 30 19:28:51.775710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.775675    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"]
Oct 30 19:28:51.785047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.784996    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq" event={"ID":"a74313a0-412d-4e3e-98f0-1bbccf4fa459","Type":"ContainerDied","Data":"b87c177aacd4c4f948d8ec8a761efcfc93f093ed95844a6c8257b3fb2043d335"}
Oct 30 19:28:51.785317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.785299    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b87c177aacd4c4f948d8ec8a761efcfc93f093ed95844a6c8257b3fb2043d335"
Oct 30 19:28:51.785522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.785509    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"
Oct 30 19:28:51.826674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.826633    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sxp2\" (UniqueName: \"kubernetes.io/projected/a74313a0-412d-4e3e-98f0-1bbccf4fa459-kube-api-access-8sxp2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:51.853801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.853154    4925 generic.go:356] "Generic (PLEG): container finished" podID="eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" containerID="3ef3ad607dba6e21b248c93ba67c0f7041e4b43f5ac1a88e56e94505ff7915d5" exitCode=0
Oct 30 19:28:51.853801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.853264    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7" event={"ID":"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3","Type":"ContainerDied","Data":"3ef3ad607dba6e21b248c93ba67c0f7041e4b43f5ac1a88e56e94505ff7915d5"}
Oct 30 19:28:51.953938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:51.953167    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:52.006711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.005920    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:28:52.250631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.247031    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-maas-api-tls\") pod \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") "
Oct 30 19:28:52.250631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.247195    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-kube-api-access-v7zc2\" (UniqueName: \"kubernetes.io/projected/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-kube-api-access-v7zc2\") pod \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") "
Oct 30 19:28:52.250631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.247360    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-openshift-service-ca\") pod \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") "
Oct 30 19:28:52.250631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.250312    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-openshift-service-ca" pod "eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" (UID: "eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:52.253214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.253188    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-kube-api-access-v7zc2" pod "eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" (UID: "eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3"). InnerVolumeSpecName "kube-api-access-v7zc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:52.253452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.253422    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-maas-api-tls" pod "eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" (UID: "eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:52.265416 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:28:52.263302    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:28:52.348946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.348908    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-metrics-certs\") pod \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\" (UID: \"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3\") "
Oct 30 19:28:52.350062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.350035    4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-openshift-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:52.350167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.350068    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-maas-api-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:52.350167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.350083    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7zc2\" (UniqueName: \"kubernetes.io/projected/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-kube-api-access-v7zc2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:52.352015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.351986    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-metrics-certs" pod "eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" (UID: "eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:52.460748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.457404    4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3-metrics-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:52.527767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.502546    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:52.948710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.944415    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7" event={"ID":"eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3","Type":"ContainerDied","Data":"316524c853b2a5111f769a9357dfff53a1b66547128947e3cfd1af42b23b7aee"}
Oct 30 19:28:52.948710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.944500    4925 scope.go:122] "RemoveContainer" containerID="3ef3ad607dba6e21b248c93ba67c0f7041e4b43f5ac1a88e56e94505ff7915d5"
Oct 30 19:28:52.948710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.944625    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"
Oct 30 19:28:52.949146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.949113    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d" event={"ID":"c3b500b6-9c01-4197-899a-3e72217112a4","Type":"ContainerStarted","Data":"3a048064d8349800a9823cd2e3f72fde5a9bcc0f80643a5b55b4c9ec1f29d242"}
Oct 30 19:28:52.950400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.950379    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:28:52.950589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.950568    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d" event={"ID":"c3b500b6-9c01-4197-899a-3e72217112a4","Type":"ContainerStarted","Data":"e4eb4bc23940046d4442530330beb79ed27f8d44c6a8a7f1a3364ad460ae444e"}
Oct 30 19:28:52.973619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.973569    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d" podStartSLOduration=3.265531984 podStartE2EDuration="3.973552389s" totalImagesPullingTime="708.020405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:52.970628222 +0000 UTC m=+1367.873641259" watchObservedRunningTime="2026-10-30 19:28:52.973552389 +0000 UTC m=+1367.876565427"
Oct 30 19:28:52.991945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:52.991917    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"]
Oct 30 19:28:53.003710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:53.000107    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a718a-7d8c596b5c-wpjl7"]
Oct 30 19:28:53.267880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:53.267791    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" path="/var/lib/kubelet/pods/eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3/volumes"
Oct 30 19:28:53.268451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:53.268428    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb39302f-43cf-420b-a305-73813e446ec0" path="/var/lib/kubelet/pods/eb39302f-43cf-420b-a305-73813e446ec0/volumes"
Oct 30 19:28:53.313259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:53.313173    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:53.923210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:53.923170    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:54.287543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:54.287451    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:54.453759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:54.453728    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:54.631523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:54.631430    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:55.623148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.623116    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"]
Oct 30 19:28:55.624784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.624760    4925 cpu_manager.go:414] "containerMap: removing container" podUID="eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" containerName="maas-api"
Oct 30 19:28:55.624942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.624926    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" containerName="maas-api"
Oct 30 19:28:55.625053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.625041    4925 cpu_manager.go:414] "containerMap: removing container" podUID="a74313a0-412d-4e3e-98f0-1bbccf4fa459" containerName="revoke-keys"
Oct 30 19:28:55.625135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.625124    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74313a0-412d-4e3e-98f0-1bbccf4fa459" containerName="revoke-keys"
Oct 30 19:28:55.628730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.625477    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf4cf20-f9a7-4ff1-ae2e-7741b49fddd3" containerName="maas-api"
Oct 30 19:28:55.628730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.626919    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74313a0-412d-4e3e-98f0-1bbccf4fa459" containerName="revoke-keys"
Oct 30 19:28:55.631572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.631550    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.654469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.654444    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-891d80-serving-cert\""
Oct 30 19:28:55.659566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.659538    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"]
Oct 30 19:28:55.758981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.755145    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde62c69-0252-4744-9d52-85185abad07d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.758981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.755265    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-maas-api-tls\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.758981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.755356    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhz7r\" (UniqueName: \"kubernetes.io/projected/bde62c69-0252-4744-9d52-85185abad07d-kube-api-access-bhz7r\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.758981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.755460    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-metrics-certs\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.858652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.858369    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:55.858859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.858820    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde62c69-0252-4744-9d52-85185abad07d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.873861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.859065    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-maas-api-tls\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.873861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.859166    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhz7r\" (UniqueName: \"kubernetes.io/projected/bde62c69-0252-4744-9d52-85185abad07d-kube-api-access-bhz7r\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.873861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.859279    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-metrics-certs\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.873861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.859796    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde62c69-0252-4744-9d52-85185abad07d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.876296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.876265    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-metrics-certs\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.883742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.881944    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-maas-api-tls\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.893970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.888089    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhz7r\" (UniqueName: \"kubernetes.io/projected/bde62c69-0252-4744-9d52-85185abad07d-kube-api-access-bhz7r\") pod \"maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:55.957740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:55.957706    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:56.057441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.057372    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:56.061201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.060059    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"]
Oct 30 19:28:56.075735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.075523    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5848a1f7-q4h9g"]
Oct 30 19:28:56.157173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.157142    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"]
Oct 30 19:28:56.162071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.162041    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:28:56.172712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.172363    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"]
Oct 30 19:28:56.177658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.175909    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f61e62-dockercfg-66n9h\""
Oct 30 19:28:56.177658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.175999    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f61e62\""
Oct 30 19:28:56.203833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.203804    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"]
Oct 30 19:28:56.223360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.223324    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"]
Oct 30 19:28:56.223532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.223425    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:28:56.243627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.243592    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:56.507766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.486963    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba76082c-f99e-47f2-8d57-0283bfb7e872-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx\" (UID: \"ba76082c-f99e-47f2-8d57-0283bfb7e872\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:28:56.507766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.487110    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a049015-a805-45ec-bbe8-786155bab0bb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f\" (UID: \"9a049015-a805-45ec-bbe8-786155bab0bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:28:56.507766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.487264    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5zd\" (UniqueName: \"kubernetes.io/projected/ba76082c-f99e-47f2-8d57-0283bfb7e872-kube-api-access-hn5zd\") pod \"payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx\" (UID: \"ba76082c-f99e-47f2-8d57-0283bfb7e872\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:28:56.507766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.487312    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bsj\" (UniqueName: \"kubernetes.io/projected/9a049015-a805-45ec-bbe8-786155bab0bb-kube-api-access-k4bsj\") pod \"payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f\" (UID: \"9a049015-a805-45ec-bbe8-786155bab0bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:28:56.597382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.595920    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"]
Oct 30 19:28:56.636067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.629495    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bsj\" (UniqueName: \"kubernetes.io/projected/9a049015-a805-45ec-bbe8-786155bab0bb-kube-api-access-k4bsj\") pod \"payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f\" (UID: \"9a049015-a805-45ec-bbe8-786155bab0bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:28:56.636067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.629771    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba76082c-f99e-47f2-8d57-0283bfb7e872-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx\" (UID: \"ba76082c-f99e-47f2-8d57-0283bfb7e872\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:28:56.636067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.629937    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a049015-a805-45ec-bbe8-786155bab0bb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f\" (UID: \"9a049015-a805-45ec-bbe8-786155bab0bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:28:56.636067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.630135    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5zd\" (UniqueName: \"kubernetes.io/projected/ba76082c-f99e-47f2-8d57-0283bfb7e872-kube-api-access-hn5zd\") pod \"payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx\" (UID: \"ba76082c-f99e-47f2-8d57-0283bfb7e872\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:28:56.662722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.656338    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba76082c-f99e-47f2-8d57-0283bfb7e872-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx\" (UID: \"ba76082c-f99e-47f2-8d57-0283bfb7e872\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:28:56.698735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.670930    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a049015-a805-45ec-bbe8-786155bab0bb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f\" (UID: \"9a049015-a805-45ec-bbe8-786155bab0bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:28:56.774934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.693126    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bsj\" (UniqueName: \"kubernetes.io/projected/9a049015-a805-45ec-bbe8-786155bab0bb-kube-api-access-k4bsj\") pod \"payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f\" (UID: \"9a049015-a805-45ec-bbe8-786155bab0bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:28:56.774934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.742028    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5zd\" (UniqueName: \"kubernetes.io/projected/ba76082c-f99e-47f2-8d57-0283bfb7e872-kube-api-access-hn5zd\") pod \"payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx\" (UID: \"ba76082c-f99e-47f2-8d57-0283bfb7e872\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:28:56.800731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.798829    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:28:56.815128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:56.815021    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:28:57.000800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:57.000745    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp" event={"ID":"bde62c69-0252-4744-9d52-85185abad07d","Type":"ContainerStarted","Data":"3823e24b93b4e976f583b182c9b2a6ea0b78c134cba40f860ed099311582539d"}
Oct 30 19:28:57.065323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:57.065254    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:57.258371 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:57.258343    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-e25d87fbe1a2483134c3b9f9f73fc7de9cf466ae2c8a55ddf048a1bf3bb677a8 WatchSource:0}: Error finding container e25d87fbe1a2483134c3b9f9f73fc7de9cf466ae2c8a55ddf048a1bf3bb677a8: Status 404 returned error can't find the container with id e25d87fbe1a2483134c3b9f9f73fc7de9cf466ae2c8a55ddf048a1bf3bb677a8
Oct 30 19:28:57.269001 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:28:57.268974    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-4dfccffb7ea9e94d2700d323153491bbc01f1c90b6d85d4a6434e2fa0b576320 WatchSource:0}: Error finding container 4dfccffb7ea9e94d2700d323153491bbc01f1c90b6d85d4a6434e2fa0b576320: Status 404 returned error can't find the container with id 4dfccffb7ea9e94d2700d323153491bbc01f1c90b6d85d4a6434e2fa0b576320
Oct 30 19:28:57.269327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:57.269299    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de2891d-1fc5-4202-892a-a79fec28a009" path="/var/lib/kubelet/pods/5de2891d-1fc5-4202-892a-a79fec28a009/volumes"
Oct 30 19:28:57.269912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:57.269849    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"]
Oct 30 19:28:57.269912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:57.269905    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"]
Oct 30 19:28:57.525782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:57.525746    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:58.023216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.023172    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx" event={"ID":"ba76082c-f99e-47f2-8d57-0283bfb7e872","Type":"ContainerStarted","Data":"46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff"}
Oct 30 19:28:58.023679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.023289    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:28:58.023679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.023314    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx" event={"ID":"ba76082c-f99e-47f2-8d57-0283bfb7e872","Type":"ContainerStarted","Data":"4dfccffb7ea9e94d2700d323153491bbc01f1c90b6d85d4a6434e2fa0b576320"}
Oct 30 19:28:58.027076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.027046    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f" event={"ID":"9a049015-a805-45ec-bbe8-786155bab0bb","Type":"ContainerStarted","Data":"3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662"}
Oct 30 19:28:58.027229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.027096    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f" event={"ID":"9a049015-a805-45ec-bbe8-786155bab0bb","Type":"ContainerStarted","Data":"e25d87fbe1a2483134c3b9f9f73fc7de9cf466ae2c8a55ddf048a1bf3bb677a8"}
Oct 30 19:28:58.027549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.027528    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:28:58.030871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.030846    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp" event={"ID":"bde62c69-0252-4744-9d52-85185abad07d","Type":"ContainerStarted","Data":"d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0"}
Oct 30 19:28:58.031011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.030990    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:28:58.043563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.043515    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx" podStartSLOduration=2.043504023 podStartE2EDuration="2.043504023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:58.041030538 +0000 UTC m=+1372.944043575" watchObservedRunningTime="2026-10-30 19:28:58.043504023 +0000 UTC m=+1372.946517108"
Oct 30 19:28:58.065995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.065941    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp" podStartSLOduration=2.557303111 podStartE2EDuration="3.065925083s" totalImagesPullingTime="508.621972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:58.059801333 +0000 UTC m=+1372.962814369" watchObservedRunningTime="2026-10-30 19:28:58.065925083 +0000 UTC m=+1372.968938122"
Oct 30 19:28:58.429712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.428837    4925 patch_prober.go:29] interesting pod/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:58.429712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.428905    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:58.572443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:58.572305    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:59.028958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:59.028889    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f" podStartSLOduration=3.028867339 podStartE2EDuration="3.028867339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:58.079858142 +0000 UTC m=+1372.982871180" watchObservedRunningTime="2026-10-30 19:28:59.028867339 +0000 UTC m=+1373.931880379"
Oct 30 19:28:59.034998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:59.034967    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"]
Oct 30 19:28:59.039738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:59.039669    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd981257-wwdsp"]
Oct 30 19:28:59.267778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:59.267744    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c02ec99-55f4-42df-9e38-702359356667" path="/var/lib/kubelet/pods/9c02ec99-55f4-42df-9e38-702359356667/volumes"
Oct 30 19:28:59.833945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:59.833902    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:59.982378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:28:59.982350    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:29:00.001010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:00.000981    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:00.343748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:00.343616    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:00.955006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:00.954166    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:01.298848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:01.298762    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:01.620525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:01.620436    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:02.049447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.049413    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"]
Oct 30 19:29:02.060600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.060571    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.075918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.075881    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8e6159-openshift-default-dockercfg-sm7cb\""
Oct 30 19:29:02.080854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.080825    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"]
Oct 30 19:29:02.084308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.084281    4925 generic.go:356] "Generic (PLEG): container finished" podID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerID="04ad9f219c6fbe81c0c26e0679841c4f2144de70c979e0ccadc6fc559a6d6234" exitCode=0
Oct 30 19:29:02.084478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.084459    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" event={"ID":"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9","Type":"ContainerDied","Data":"04ad9f219c6fbe81c0c26e0679841c4f2144de70c979e0ccadc6fc559a6d6234"}
Oct 30 19:29:02.085071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.085040    4925 scope.go:122] "RemoveContainer" containerID="04ad9f219c6fbe81c0c26e0679841c4f2144de70c979e0ccadc6fc559a6d6234"
Oct 30 19:29:02.087310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.087283    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:02.087395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.087356    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:02.087395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.087388    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:02.130617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.130592    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-credential-socket\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.130739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.130630    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-socket\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.130788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.130740    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istiod-ca-cert\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.130841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.130826    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-token\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.131099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.130889    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-certs\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.131099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.130953    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-data\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.131099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.131012    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzckb\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-kube-api-access-bzckb\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.131099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.131063    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-podinfo\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.131099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.131089    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-envoy\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.169207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.169173    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:02.232547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.232506    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-certs\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.232789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.232617    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-data\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.232789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.232744    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzckb\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-kube-api-access-bzckb\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.232904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.232819    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-podinfo\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.232904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.232860    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-envoy\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.232981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.232926    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-credential-socket\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.232981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.232948    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-certs\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.232981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.232961    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-socket\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.233209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.233189    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-data\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.233758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.233681    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-socket\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.234244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.233963    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-credential-socket\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.234244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.234090    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istiod-ca-cert\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.234244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.234225    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-token\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.235140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.235115    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istiod-ca-cert\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.236150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.236127    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-podinfo\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.241556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.241527    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-envoy\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.261714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.261576    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzckb\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-kube-api-access-bzckb\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.314758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.314306    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-token\") pod \"e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") " pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.390713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.380196    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:02.421722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.406957    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:02.959914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.959882    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"]
Oct 30 19:29:02.962226 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:29:02.962197    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb1b4ae_9bbe_4202_aeac_28f45cdb7ae1.slice/crio-d846d3a433692c134689bdda0687165f0e140658ef49b387e4ddc0db7fc51daf WatchSource:0}: Error finding container d846d3a433692c134689bdda0687165f0e140658ef49b387e4ddc0db7fc51daf: Status 404 returned error can't find the container with id d846d3a433692c134689bdda0687165f0e140658ef49b387e4ddc0db7fc51daf
Oct 30 19:29:02.965457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.965379    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:02.965568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.965499    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:02.965568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:02.965543    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:03.042910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:03.032234    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:03.099548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:03.099514    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" event={"ID":"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1","Type":"ContainerStarted","Data":"d846d3a433692c134689bdda0687165f0e140658ef49b387e4ddc0db7fc51daf"}
Oct 30 19:29:03.103345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:03.103293    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" event={"ID":"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9","Type":"ContainerStarted","Data":"857dd413838715c45ae16ebe278b430634597542facb04734b9ae911a3ec3a0b"}
Oct 30 19:29:03.790416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:03.789919    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:03.975877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:03.975843    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:29:03.976366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:03.975982    4925 patch_prober.go:29] interesting pod/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:29:03.976366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:03.976031    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:29:04.047256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:04.047185    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:29:04.134893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:04.134831    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" event={"ID":"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1","Type":"ContainerStarted","Data":"22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458"}
Oct 30 19:29:04.168299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:04.168245    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" podStartSLOduration=2.168230672 podStartE2EDuration="2.168230672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:04.165218619 +0000 UTC m=+1379.068231655" watchObservedRunningTime="2026-10-30 19:29:04.168230672 +0000 UTC m=+1379.071243708"
Oct 30 19:29:04.427139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:04.427111    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:04.441198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:04.441168    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:04.788611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:04.788527    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:04.976098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:04.976063    4925 patch_prober.go:29] interesting pod/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:29:04.976550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:04.976124    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:29:05.143182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:05.143149    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:05.144255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:05.144233    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:29:05.553509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:05.553418    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:05.975929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:05.975894    4925 patch_prober.go:29] interesting pod/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:29:05.976126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:05.975977    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:29:06.587280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:06.587249    4925 scope.go:122] "RemoveContainer" containerID="a1a6f8b2ef555e4571330eb950890ed2dffe8ba8f2c3429426290169d08a40a6"
Oct 30 19:29:06.623922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:06.623901    4925 scope.go:122] "RemoveContainer" containerID="536e05a56d5cfc76187a44ac54d238dc6c78dc8f0885ae8da029281ab3794dd8"
Oct 30 19:29:06.634527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:06.634503    4925 scope.go:122] "RemoveContainer" containerID="b9012b46285a6b39c79b2ebef9e01129ac70627389b9cf574e3a9b56f6a7f7ed"
Oct 30 19:29:06.644165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:06.644150    4925 scope.go:122] "RemoveContainer" containerID="a3aa9d40ede6336ffce84fdaa6a0b7f90e190e019e44dc1ed071ed5b77f74cb1"
Oct 30 19:29:06.644624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:06.644600    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:06.901218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:06.900298    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:06.994723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:06.985478    4925 patch_prober.go:29] interesting pod/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:29:06.994723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:06.985552    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:29:07.752076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:07.752040    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:07.938233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:07.938194    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:07.976794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:07.976760    4925 patch_prober.go:29] interesting pod/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:29:07.976982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:07.976815    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:29:08.140787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:08.111561    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:08.774016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:08.773909    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:08.870720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:08.869752    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"]
Oct 30 19:29:08.898744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:08.894255    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48b1a6a7-t7849"]
Oct 30 19:29:08.945016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:08.943199    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:08.978366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:08.976781    4925 patch_prober.go:29] interesting pod/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 19:29:08.978366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:08.976842    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 19:29:09.069029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:09.068941    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:29:09.082714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:09.076851    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:29:09.143712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:09.134018    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:09.280715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:09.278517    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02488c1-623d-4da3-86d3-9141f35072b6" path="/var/lib/kubelet/pods/f02488c1-623d-4da3-86d3-9141f35072b6/volumes"
Oct 30 19:29:09.420709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:09.420379    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:09.981041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:09.981017    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:29:09.981574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:09.981524    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:29:09.982275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:09.982255    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:29:10.029372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:10.029327    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"]
Oct 30 19:29:10.040159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:10.040128    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d0c1c42-7rxh4"]
Oct 30 19:29:10.514225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:10.514192    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:10.795397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:10.795312    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"]
Oct 30 19:29:10.807894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:10.807862    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"]
Oct 30 19:29:10.925306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:10.925263    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:11.098226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:11.098135    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:11.111710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:11.106737    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"]
Oct 30 19:29:11.115715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:11.112438    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57874b60-n9z8g"]
Oct 30 19:29:11.274763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:11.274719    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e39e4f-2be6-4c2e-a696-f01471a5140f" path="/var/lib/kubelet/pods/02e39e4f-2be6-4c2e-a696-f01471a5140f/volumes"
Oct 30 19:29:11.277043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:11.277010    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b74efc3-2fa8-4460-98a5-4bf5c1c7223f" path="/var/lib/kubelet/pods/9b74efc3-2fa8-4460-98a5-4bf5c1c7223f/volumes"
Oct 30 19:29:11.314388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:11.314296    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:12.264594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:12.264557    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:12.929736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:12.928875    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:14.013652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:14.013617    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:14.182125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:14.182093    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:14.295662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:14.295570    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"]
Oct 30 19:29:14.460590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:14.460559    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:14.650421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:14.650387    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:15.067537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:15.067429    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:15.250940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:15.250905    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:15.645664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:15.645623    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:15.796287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:15.796248    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc" podUID="8c0a7656-3a0d-4cd4-a250-80218d696f48" containerName="payload-processing" containerID="cri-o://8a59e214d3afaae0e4408fde016bf3dd690f280af85b4bab6070121f5862c6de" gracePeriod=25
Oct 30 19:29:15.808678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:15.808645    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5" podUID="effed5a1-a05b-4955-aca9-3ca8a882c094" containerName="payload-pre-processing" containerID="cri-o://77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae" gracePeriod=25
Oct 30 19:29:16.005733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.004307    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:16.446098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.446068    4925 generic.go:356] "Generic (PLEG): container finished" podID="8c0a7656-3a0d-4cd4-a250-80218d696f48" containerID="8a59e214d3afaae0e4408fde016bf3dd690f280af85b4bab6070121f5862c6de" exitCode=0
Oct 30 19:29:16.446476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.446218    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc" event={"ID":"8c0a7656-3a0d-4cd4-a250-80218d696f48","Type":"ContainerDied","Data":"8a59e214d3afaae0e4408fde016bf3dd690f280af85b4bab6070121f5862c6de"}
Oct 30 19:29:16.447908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.447888    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:29:16.448019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.447921    4925 generic.go:356] "Generic (PLEG): container finished" podID="effed5a1-a05b-4955-aca9-3ca8a882c094" containerID="77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae" exitCode=0
Oct 30 19:29:16.448145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.448122    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5" event={"ID":"effed5a1-a05b-4955-aca9-3ca8a882c094","Type":"ContainerDied","Data":"77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae"}
Oct 30 19:29:16.448219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.448167    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5" event={"ID":"effed5a1-a05b-4955-aca9-3ca8a882c094","Type":"ContainerDied","Data":"4969bdbc852bd282a763492a59a40890f7e6e2231797a757e4a5f84bde6ae33a"}
Oct 30 19:29:16.448219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.448197    4925 scope.go:122] "RemoveContainer" containerID="77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae"
Oct 30 19:29:16.467699 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.467665    4925 scope.go:122] "RemoveContainer" containerID="77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae"
Oct 30 19:29:16.468025 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:29:16.467999    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae\": container with ID starting with 77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae not found: ID does not exist" containerID="77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae"
Oct 30 19:29:16.468123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.468037    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae"} err="failed to get container status \"77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae\": rpc error: code = NotFound desc = could not find container \"77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae\": container with ID starting with 77413012e9f6132b1605fe486e1a8f9286895eea40481cb5a7a4597acd33daae not found: ID does not exist"
Oct 30 19:29:16.489657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.489628    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:29:16.502227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.502193    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effed5a1-a05b-4955-aca9-3ca8a882c094-kube-api-access-8zl9f\" (UniqueName: \"kubernetes.io/projected/effed5a1-a05b-4955-aca9-3ca8a882c094-kube-api-access-8zl9f\") pod \"effed5a1-a05b-4955-aca9-3ca8a882c094\" (UID: \"effed5a1-a05b-4955-aca9-3ca8a882c094\") "
Oct 30 19:29:16.503202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.503176    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effed5a1-a05b-4955-aca9-3ca8a882c094-plugins-config-volume" pod "effed5a1-a05b-4955-aca9-3ca8a882c094" (UID: "effed5a1-a05b-4955-aca9-3ca8a882c094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:16.503336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.503311    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effed5a1-a05b-4955-aca9-3ca8a882c094-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effed5a1-a05b-4955-aca9-3ca8a882c094-plugins-config-volume\") pod \"effed5a1-a05b-4955-aca9-3ca8a882c094\" (UID: \"effed5a1-a05b-4955-aca9-3ca8a882c094\") "
Oct 30 19:29:16.504235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.504211    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effed5a1-a05b-4955-aca9-3ca8a882c094-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:16.505069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.505048    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effed5a1-a05b-4955-aca9-3ca8a882c094-kube-api-access-8zl9f" pod "effed5a1-a05b-4955-aca9-3ca8a882c094" (UID: "effed5a1-a05b-4955-aca9-3ca8a882c094"). InnerVolumeSpecName "kube-api-access-8zl9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:16.525485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.525396    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"]
Oct 30 19:29:16.530631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.527505    4925 cpu_manager.go:414] "containerMap: removing container" podUID="effed5a1-a05b-4955-aca9-3ca8a882c094" containerName="payload-pre-processing"
Oct 30 19:29:16.530631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.527531    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effed5a1-a05b-4955-aca9-3ca8a882c094" containerName="payload-pre-processing"
Oct 30 19:29:16.530631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.527564    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8c0a7656-3a0d-4cd4-a250-80218d696f48" containerName="payload-processing"
Oct 30 19:29:16.530631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.527572    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0a7656-3a0d-4cd4-a250-80218d696f48" containerName="payload-processing"
Oct 30 19:29:16.530631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.527858    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="effed5a1-a05b-4955-aca9-3ca8a882c094" containerName="payload-pre-processing"
Oct 30 19:29:16.530631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.527882    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0a7656-3a0d-4cd4-a250-80218d696f48" containerName="payload-processing"
Oct 30 19:29:16.554025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.553990    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.559818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.559788    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"]
Oct 30 19:29:16.581258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.581225    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-afdf16-openshift-default-dockercfg-x4cz2\""
Oct 30 19:29:16.605152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605123    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c0a7656-3a0d-4cd4-a250-80218d696f48-kube-api-access-n78pb\" (UniqueName: \"kubernetes.io/projected/8c0a7656-3a0d-4cd4-a250-80218d696f48-kube-api-access-n78pb\") pod \"8c0a7656-3a0d-4cd4-a250-80218d696f48\" (UID: \"8c0a7656-3a0d-4cd4-a250-80218d696f48\") "
Oct 30 19:29:16.605344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605197    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0a7656-3a0d-4cd4-a250-80218d696f48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0a7656-3a0d-4cd4-a250-80218d696f48-plugins-config-volume\") pod \"8c0a7656-3a0d-4cd4-a250-80218d696f48\" (UID: \"8c0a7656-3a0d-4cd4-a250-80218d696f48\") "
Oct 30 19:29:16.605660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605626    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-token\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.605811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605753    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-data\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.605884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605817    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpkw9\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-kube-api-access-xpkw9\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.605884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605866    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-envoy\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.606008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605955    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istiod-ca-cert\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.606008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605966    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0a7656-3a0d-4cd4-a250-80218d696f48-plugins-config-volume" pod "8c0a7656-3a0d-4cd4-a250-80218d696f48" (UID: "8c0a7656-3a0d-4cd4-a250-80218d696f48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:16.606008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605980    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-credential-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.606008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.605998    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-certs\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.606381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.606356    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.606468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.606419    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-podinfo\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.606765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.606748    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0a7656-3a0d-4cd4-a250-80218d696f48-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:16.606847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.606769    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zl9f\" (UniqueName: \"kubernetes.io/projected/effed5a1-a05b-4955-aca9-3ca8a882c094-kube-api-access-8zl9f\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:16.619882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.619856    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0a7656-3a0d-4cd4-a250-80218d696f48-kube-api-access-n78pb" pod "8c0a7656-3a0d-4cd4-a250-80218d696f48" (UID: "8c0a7656-3a0d-4cd4-a250-80218d696f48"). InnerVolumeSpecName "kube-api-access-n78pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:16.680711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.679149    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:16.707977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.707930    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istiod-ca-cert\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.708159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.707989    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-credential-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.708159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.708017    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-certs\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.708289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.708191    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.708289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.708227    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-podinfo\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.708450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.708433    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-token\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.708537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.708523    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-data\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.708617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.708603    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpkw9\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-kube-api-access-xpkw9\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.708677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.708667    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-envoy\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.708845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.708830    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n78pb\" (UniqueName: \"kubernetes.io/projected/8c0a7656-3a0d-4cd4-a250-80218d696f48-kube-api-access-n78pb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:16.710467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.710439    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istiod-ca-cert\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.710959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.710938    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-data\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.711126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.711062    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-certs\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.711232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.711037    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-credential-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.711232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.711200    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-socket\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.712248 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.712227    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-envoy\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.714096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.714071    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-podinfo\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.723426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.723396    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-token\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.724369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.724346    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpkw9\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-kube-api-access-xpkw9\") pod \"e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") " pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:16.876769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:16.876723    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:17.232155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.232121    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"]
Oct 30 19:29:17.232592 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:29:17.232565    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb24c7dd_850b_40a4_b25b_2450e5fffe39.slice/crio-aa288202ca014c5e6143068f27e0e38b099bb6bb283328176916c08d6d1e0975 WatchSource:0}: Error finding container aa288202ca014c5e6143068f27e0e38b099bb6bb283328176916c08d6d1e0975: Status 404 returned error can't find the container with id aa288202ca014c5e6143068f27e0e38b099bb6bb283328176916c08d6d1e0975
Oct 30 19:29:17.236035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.235997    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:17.236114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.236086    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:17.236155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.236128    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:17.460170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.460136    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"
Oct 30 19:29:17.460879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.460784    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc" event={"ID":"8c0a7656-3a0d-4cd4-a250-80218d696f48","Type":"ContainerDied","Data":"b308b85edb3b143d71c21a0ad928894edbe547dc991f17a645e35e8eb462ade0"}
Oct 30 19:29:17.460879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.460850    4925 scope.go:122] "RemoveContainer" containerID="8a59e214d3afaae0e4408fde016bf3dd690f280af85b4bab6070121f5862c6de"
Oct 30 19:29:17.496121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.494741    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"
Oct 30 19:29:17.521985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.521949    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" event={"ID":"fb24c7dd-850b-40a4-b25b-2450e5fffe39","Type":"ContainerStarted","Data":"eacfe6ef81a7811d3b8141d985b42a866e9dcccf04200f89c89022dd13d2dce1"}
Oct 30 19:29:17.522173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.522019    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" event={"ID":"fb24c7dd-850b-40a4-b25b-2450e5fffe39","Type":"ContainerStarted","Data":"aa288202ca014c5e6143068f27e0e38b099bb6bb283328176916c08d6d1e0975"}
Oct 30 19:29:17.522173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.522123    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:17.553736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.539747    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"]
Oct 30 19:29:17.553736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.544046    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b2e27-56c4c6f766-fphsc"]
Oct 30 19:29:17.597825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.597791    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"]
Oct 30 19:29:17.615759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.615735    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b2e27-5b998f9f9-8f9j5"]
Oct 30 19:29:17.617735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.616295    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podStartSLOduration=3.61628265 podStartE2EDuration="3.61628265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:17.60680463 +0000 UTC m=+1392.509817669" watchObservedRunningTime="2026-10-30 19:29:17.61628265 +0000 UTC m=+1392.519295686"
Oct 30 19:29:17.879885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:17.879794    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:18.414665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:18.414631    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:18.884397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:18.884321    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:19.141415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:19.140177    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"]
Oct 30 19:29:19.231711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:19.216457    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df9cf0-swvlr"]
Oct 30 19:29:19.278791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:19.274009    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4359741e-e0c0-4204-b437-2d5a4147f65d" path="/var/lib/kubelet/pods/4359741e-e0c0-4204-b437-2d5a4147f65d/volumes"
Oct 30 19:29:19.278791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:19.274827    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0a7656-3a0d-4cd4-a250-80218d696f48" path="/var/lib/kubelet/pods/8c0a7656-3a0d-4cd4-a250-80218d696f48/volumes"
Oct 30 19:29:19.278791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:19.275453    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effed5a1-a05b-4955-aca9-3ca8a882c094" path="/var/lib/kubelet/pods/effed5a1-a05b-4955-aca9-3ca8a882c094/volumes"
Oct 30 19:29:19.545357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:19.545264    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:19.546272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:19.546249    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:29:19.637095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:19.637063    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:19.990260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:19.990230    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:20.156244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:20.156204    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:20.577529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:20.577497    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:21.034931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:21.034897    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"]
Oct 30 19:29:21.041960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:21.041938    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06082bd8-mdpnf"]
Oct 30 19:29:21.197405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:21.197373    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:21.268286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:21.268254    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0378d84f-51a8-46fe-bd81-6976fa1ef734" path="/var/lib/kubelet/pods/0378d84f-51a8-46fe-bd81-6976fa1ef734/volumes"
Oct 30 19:29:21.406712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:21.402187    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:22.128422 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:29:22.128380    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:29:22.238643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:22.238611    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:23.733726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:23.733665    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:24.378598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:24.378562    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:24.952777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:24.952737    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:25.049871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.022571    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:29:25.049871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.023222    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:29:25.155992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.155921    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:25.156914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.156888    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:25.407718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.394843    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:25.550952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.550890    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" podUID="eafe5212-0e46-4f23-b375-848434828400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:25.784895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.784792    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:25.795553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.795509    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:25.957872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.957823    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Oct 30 19:29:25.983715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.958475    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:26.018714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.958543    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:26.019670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:25.994647    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:26.090563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.005752    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:26.095032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.005790    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:26.100047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.032920    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:26.123609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.123579    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:29:26.159444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.159400    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:26.199824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.199780    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:26.314155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.314128    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:29:26.561052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.561007    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" podUID="eafe5212-0e46-4f23-b375-848434828400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded"
Oct 30 19:29:26.561930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.561904    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:29:26.690232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.690203    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:26.804326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.804284    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:29:26.811736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:26.805844    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:29:27.127390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:27.127271    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.128664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:27.128641    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:29:27.154974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:27.154942    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:27.184715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:27.162762    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.925713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:27.908859    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:29:27.939226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:27.939192    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:27.952850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:27.952809    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.958714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:27.957788    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.968293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:27.968247    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.005514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.005469    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.057985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.057935    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.067097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.067061    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:29:28.128715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.126649    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:28.132757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.129770    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.133019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.132992    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.376772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.359931    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.545839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.545785    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" podUID="eafe5212-0e46-4f23-b375-848434828400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.624074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.624034    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.881723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.878750    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.908397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.908358    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.911729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.909771    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:29:28.911729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.910196    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:29:28.963637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.963606    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:28.964070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.964049    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": context deadline exceeded"
Oct 30 19:29:28.972808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.972763    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.973317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:28.973225    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:29:29.091723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.063231    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.092300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.065041    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:29:29.092526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.092508    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:29:29.178792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.166897    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.178792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.175566    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:29:29.178792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.177831    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:29:29.205720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.187577    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.347832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.322941    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" podUID="eafe5212-0e46-4f23-b375-848434828400" containerName="main" containerID="cri-o://a638b417eed1cc7f0bcd0f72a335aede6aae3f108956401a7388fbed888a83f7" gracePeriod=45
Oct 30 19:29:29.373259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.360755    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.477800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.471310    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:29:29.477800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.471367    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:29:29.477800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.471399    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:29:29.630712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.626057    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:29:29.711785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.706978    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:29:29.785779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.785408    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" podUID="eafe5212-0e46-4f23-b375-848434828400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": dial tcp 10.132.0.123:8000: connect: connection refused"
Oct 30 19:29:29.921130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.921071    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:29:29.933764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.932781    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:29:29.933764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.933323    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.954778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.954732    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:29:29.959303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:29.959267    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:30.134784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.122128    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:29:30.210679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.210643    4925 generic.go:356] "Generic (PLEG): container finished" podID="eafe5212-0e46-4f23-b375-848434828400" containerID="a638b417eed1cc7f0bcd0f72a335aede6aae3f108956401a7388fbed888a83f7" exitCode=0
Oct 30 19:29:30.212762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.212732    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" event={"ID":"eafe5212-0e46-4f23-b375-848434828400","Type":"ContainerDied","Data":"a638b417eed1cc7f0bcd0f72a335aede6aae3f108956401a7388fbed888a83f7"}
Oct 30 19:29:30.477863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.477833    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:29:30.489192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.489148    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:29:30.583739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.583275    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:29:30.589895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.588131    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:29:30.590934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.590914    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:29:30.689189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.689155    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-model-cache\") pod \"eafe5212-0e46-4f23-b375-848434828400\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") "
Oct 30 19:29:30.689415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.689390    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafe5212-0e46-4f23-b375-848434828400-tls-certs\" (UniqueName: \"kubernetes.io/secret/eafe5212-0e46-4f23-b375-848434828400-tls-certs\") pod \"eafe5212-0e46-4f23-b375-848434828400\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") "
Oct 30 19:29:30.689743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.689673    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-model-cache" pod "eafe5212-0e46-4f23-b375-848434828400" (UID: "eafe5212-0e46-4f23-b375-848434828400"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:30.689865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.689733    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-home\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-home\") pod \"eafe5212-0e46-4f23-b375-848434828400\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") "
Oct 30 19:29:30.689962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.689949    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-tmp-dir\") pod \"eafe5212-0e46-4f23-b375-848434828400\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") "
Oct 30 19:29:30.690973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.690172    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-home" pod "eafe5212-0e46-4f23-b375-848434828400" (UID: "eafe5212-0e46-4f23-b375-848434828400"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:30.690973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.690520    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-tmp-dir" pod "eafe5212-0e46-4f23-b375-848434828400" (UID: "eafe5212-0e46-4f23-b375-848434828400"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:30.691216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.691198    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-dshm\") pod \"eafe5212-0e46-4f23-b375-848434828400\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") "
Oct 30 19:29:30.691508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.691364    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafe5212-0e46-4f23-b375-848434828400-kube-api-access-g9kk6\" (UniqueName: \"kubernetes.io/projected/eafe5212-0e46-4f23-b375-848434828400-kube-api-access-g9kk6\") pod \"eafe5212-0e46-4f23-b375-848434828400\" (UID: \"eafe5212-0e46-4f23-b375-848434828400\") "
Oct 30 19:29:30.692573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.692552    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:30.692737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.692723    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:30.692835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.692820    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:30.693603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.693533    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafe5212-0e46-4f23-b375-848434828400-tls-certs" pod "eafe5212-0e46-4f23-b375-848434828400" (UID: "eafe5212-0e46-4f23-b375-848434828400"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:30.729339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.722353    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafe5212-0e46-4f23-b375-848434828400-kube-api-access-g9kk6" pod "eafe5212-0e46-4f23-b375-848434828400" (UID: "eafe5212-0e46-4f23-b375-848434828400"). InnerVolumeSpecName "kube-api-access-g9kk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:30.762728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.762676    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-dshm" pod "eafe5212-0e46-4f23-b375-848434828400" (UID: "eafe5212-0e46-4f23-b375-848434828400"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:30.822721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.807865    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9kk6\" (UniqueName: \"kubernetes.io/projected/eafe5212-0e46-4f23-b375-848434828400-kube-api-access-g9kk6\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:30.822721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.807905    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafe5212-0e46-4f23-b375-848434828400-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:30.822721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.807923    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafe5212-0e46-4f23-b375-848434828400-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:30.868344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:30.868285    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:31.254425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:31.254398    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"
Oct 30 19:29:31.254883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:31.254421    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" event={"ID":"eafe5212-0e46-4f23-b375-848434828400","Type":"ContainerDied","Data":"f635afb80c6006727840dd7f61e3e42ce0a47d2159b3728e9479ca65c5f0dff0"}
Oct 30 19:29:31.254883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:31.254472    4925 scope.go:122] "RemoveContainer" containerID="a638b417eed1cc7f0bcd0f72a335aede6aae3f108956401a7388fbed888a83f7"
Oct 30 19:29:31.279256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:31.279232    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"]
Oct 30 19:29:31.283312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:31.283291    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q"]
Oct 30 19:29:31.488109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:31.487963    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:31.551902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:31.551794    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-48b1a6a7/e2e-lifecycle-model-48b1a6a7-kserve-8594699cf7-ssq6q" podUID="eafe5212-0e46-4f23-b375-848434828400" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:32.360719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:32.358393    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:33.010440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:33.010402    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:33.268186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:33.268085    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafe5212-0e46-4f23-b375-848434828400" path="/var/lib/kubelet/pods/eafe5212-0e46-4f23-b375-848434828400/volumes"
Oct 30 19:29:33.868318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:33.868275    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:34.473591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:34.473306    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:35.247712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:35.245173    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:36.168860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:36.168821    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:37.001710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:36.990134    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:29:37.001710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:36.990203    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:29:37.054709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.053800    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:37.066711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.060734    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"]
Oct 30 19:29:37.069733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.067886    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"]
Oct 30 19:29:37.079800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.079733    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26db3c94-8qhzr"]
Oct 30 19:29:37.085285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.085133    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d2abb4-4q7wx"]
Oct 30 19:29:37.267218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.267190    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf" path="/var/lib/kubelet/pods/2684b4ec-8fdd-4ade-b8fb-fabfc8f4b5cf/volumes"
Oct 30 19:29:37.267938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.267915    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e815abb7-3ccd-45af-8808-e29d808b3572" path="/var/lib/kubelet/pods/e815abb7-3ccd-45af-8808-e29d808b3572/volumes"
Oct 30 19:29:37.324936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.324910    4925 generic.go:356] "Generic (PLEG): container finished" podID="febce7a9-ce87-4830-9d20-9ace649530e2" containerID="6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b" exitCode=0
Oct 30 19:29:37.325080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.324980    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" event={"ID":"febce7a9-ce87-4830-9d20-9ace649530e2","Type":"ContainerDied","Data":"6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b"}
Oct 30 19:29:37.325454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.325431    4925 scope.go:122] "RemoveContainer" containerID="6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b"
Oct 30 19:29:37.327000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.326979    4925 generic.go:356] "Generic (PLEG): container finished" podID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerID="a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d" exitCode=0
Oct 30 19:29:37.327116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.327038    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" event={"ID":"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd","Type":"ContainerDied","Data":"a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d"}
Oct 30 19:29:37.327358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.327327    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:37.327400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.327387    4925 scope.go:122] "RemoveContainer" containerID="a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d"
Oct 30 19:29:37.327434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.327402    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:37.327471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.327435    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:37.331273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.331234    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:37.331413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.331387    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:37.331512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:37.331491    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:38.078627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.075645    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:38.359880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.359793    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" event={"ID":"febce7a9-ce87-4830-9d20-9ace649530e2","Type":"ContainerStarted","Data":"7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064"}
Oct 30 19:29:38.362749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.362720    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" event={"ID":"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd","Type":"ContainerStarted","Data":"a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920"}
Oct 30 19:29:38.581599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.581572    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:29:38.581817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.581785    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:29:38.581869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.581856    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:29:38.685418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.685390    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:29:38.685674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.685652    4925 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:29:38.685768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.685735    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:29:38.939766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:38.939278    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:39.473609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:39.473574    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:39.585712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:39.585112    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:29:39.585712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:39.585186    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:29:39.656724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:39.653000    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:39.689149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:39.689094    4925 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:29:39.689337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:39.689218    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:29:40.281784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:40.281742    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:40.528717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:40.524633    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:40.596797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:40.591404    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:29:40.596797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:40.591487    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:29:40.685366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:40.685336    4925 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:29:40.685548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:40.685402    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:29:41.357372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:41.357338    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:41.581662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:41.581626    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:29:41.582076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:41.581675    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:29:41.685218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:41.685187    4925 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:29:41.685393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:41.685235    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:29:42.354192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:42.354158    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:42.581977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:42.581946    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:29:42.582409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:42.582006    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:29:42.685270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:42.685237    4925 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:29:42.685438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:42.685292    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:29:43.376287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:43.376253    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:43.581788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:43.581756    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:29:43.581964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:43.581826    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:29:43.685671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:43.685637    4925 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:29:43.686094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:43.685764    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:29:44.455108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:44.454891    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:44.707377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:44.707287    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:29:44.708558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:44.708524    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:29:44.713514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:44.713497    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:29:44.769043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:44.769012    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:29:44.769608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:44.769585    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:29:44.776125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:44.770570    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:29:45.026131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:45.026056    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:45.197555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:45.197522    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:45.485879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:45.485838    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:45.657045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:45.656988    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:45.882356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:45.882267    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:46.933380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:46.933347    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:47.649415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:47.649370    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:48.384393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:48.383801    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:49.147136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:49.147070    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:50.047561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:50.047528    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:50.662267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:50.660828    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:50.744049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:50.744003    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"]
Oct 30 19:29:50.744269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:50.744245    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy" containerID="cri-o://ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d" gracePeriod=30
Oct 30 19:29:51.569287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:51.569239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:51.774547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:51.774220    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"]
Oct 30 19:29:51.884574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:51.884408    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:52.074712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:52.057271    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:52.321578 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:29:52.321528    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:29:52.443710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:52.408598    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:52.908946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:52.908859    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:53.654713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:53.653950    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:53.831087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:53.831057    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:54.043770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:54.041299    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:54.618980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:54.618943    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:54.956352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:54.956309    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:55.429715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:55.428227    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:56.004078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.003792    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:56.145890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.145860    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:29:56.194418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.194356    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:56.195914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.195885    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"]
Oct 30 19:29:56.197613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.197585    4925 cpu_manager.go:414] "containerMap: removing container" podUID="eafe5212-0e46-4f23-b375-848434828400" containerName="main"
Oct 30 19:29:56.197613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.197611    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafe5212-0e46-4f23-b375-848434828400" containerName="main"
Oct 30 19:29:56.197863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.197648    4925 cpu_manager.go:414] "containerMap: removing container" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy"
Oct 30 19:29:56.197863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.197659    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy"
Oct 30 19:29:56.197974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.197902    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerName="istio-proxy"
Oct 30 19:29:56.197974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.197932    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafe5212-0e46-4f23-b375-848434828400" containerName="main"
Oct 30 19:29:56.201699 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.201667    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.203404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.203378    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-certs\") pod \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") "
Oct 30 19:29:56.203516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.203481    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-credential-socket\") pod \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") "
Oct 30 19:29:56.203658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.203636    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-podinfo\") pod \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") "
Oct 30 19:29:56.203823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.203792    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-data\") pod \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") "
Oct 30 19:29:56.203891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.203866    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-socket\") pod \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") "
Oct 30 19:29:56.203954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.203900    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-envoy\") pod \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") "
Oct 30 19:29:56.204013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.203982    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-token\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-token\") pod \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") "
Oct 30 19:29:56.204075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.204028    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-kube-api-access-qk88n\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-kube-api-access-qk88n\") pod \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") "
Oct 30 19:29:56.204075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.204055    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istiod-ca-cert\") pod \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\" (UID: \"03d906b5-98ed-42f0-8ea7-3ca948f57e7b\") "
Oct 30 19:29:56.204524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.204504    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-data" pod "03d906b5-98ed-42f0-8ea7-3ca948f57e7b" (UID: "03d906b5-98ed-42f0-8ea7-3ca948f57e7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:56.204882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.204862    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-certs" pod "03d906b5-98ed-42f0-8ea7-3ca948f57e7b" (UID: "03d906b5-98ed-42f0-8ea7-3ca948f57e7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:56.205196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.205180    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-credential-socket" pod "03d906b5-98ed-42f0-8ea7-3ca948f57e7b" (UID: "03d906b5-98ed-42f0-8ea7-3ca948f57e7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:56.206507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.206459    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.206507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.206493    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.206507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.206508    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.206881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.206858    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istiod-ca-cert" pod "03d906b5-98ed-42f0-8ea7-3ca948f57e7b" (UID: "03d906b5-98ed-42f0-8ea7-3ca948f57e7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:56.207153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.207130    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-socket" pod "03d906b5-98ed-42f0-8ea7-3ca948f57e7b" (UID: "03d906b5-98ed-42f0-8ea7-3ca948f57e7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:56.209741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.208491    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-podinfo" pod "03d906b5-98ed-42f0-8ea7-3ca948f57e7b" (UID: "03d906b5-98ed-42f0-8ea7-3ca948f57e7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:29:56.209741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.209097    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ef5dca-openshift-default-dockercfg-vkm6l\""
Oct 30 19:29:56.209909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.209806    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-envoy" pod "03d906b5-98ed-42f0-8ea7-3ca948f57e7b" (UID: "03d906b5-98ed-42f0-8ea7-3ca948f57e7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:56.212371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.212349    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-token" pod "03d906b5-98ed-42f0-8ea7-3ca948f57e7b" (UID: "03d906b5-98ed-42f0-8ea7-3ca948f57e7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:56.212473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.212433    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-kube-api-access-qk88n" pod "03d906b5-98ed-42f0-8ea7-3ca948f57e7b" (UID: "03d906b5-98ed-42f0-8ea7-3ca948f57e7b"). InnerVolumeSpecName "kube-api-access-qk88n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:56.212544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.212512    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"]
Oct 30 19:29:56.308531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.308494    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-data\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.308748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.308638    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0d956-67cf-40d2-a0b0-427786ce6161-istiod-ca-cert\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.308839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.308782    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-envoy\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.308839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.308830    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.309200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.309174    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-certs\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.309290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.309229    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z59hm\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-kube-api-access-z59hm\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.309615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.309497    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-credential-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.309760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.309646    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-podinfo\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.309837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.309789    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-token\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.309960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.309944    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.310555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.310535    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.310653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.310558    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.310653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.310573    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.310653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.310586    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk88n\" (UniqueName: \"kubernetes.io/projected/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-kube-api-access-qk88n\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.310653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.310601    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d906b5-98ed-42f0-8ea7-3ca948f57e7b-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.412370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.412337    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0d956-67cf-40d2-a0b0-427786ce6161-istiod-ca-cert\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.412564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.412441    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-envoy\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.412564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.412473    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.412757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.412582    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-certs\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.412757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.412619    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z59hm\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-kube-api-access-z59hm\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.412757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.412671    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-credential-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.412925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.412836    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-podinfo\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.412925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.412911    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.413184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.413159    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-certs\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.413273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.412917    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-token\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.413273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.413187    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0d956-67cf-40d2-a0b0-427786ce6161-istiod-ca-cert\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.413273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.413244    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-credential-socket\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.413379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.413312    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-data\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.413826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.413785    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-data\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.415550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.415530    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-envoy\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.416178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.416150    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-podinfo\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.424960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.424930    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-token\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.425166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.425149    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z59hm\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-kube-api-access-z59hm\") pod \"e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") " pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.545202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.545166    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:56.768952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.768870    4925 generic.go:356] "Generic (PLEG): container finished" podID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" containerID="ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d" exitCode=0
Oct 30 19:29:56.768952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.768931    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"
Oct 30 19:29:56.769194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.768964    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" event={"ID":"03d906b5-98ed-42f0-8ea7-3ca948f57e7b","Type":"ContainerDied","Data":"ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d"}
Oct 30 19:29:56.769194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.769039    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv" event={"ID":"03d906b5-98ed-42f0-8ea7-3ca948f57e7b","Type":"ContainerDied","Data":"2d1ff58f12e747587f004d5289014f93e16acca81ac69556df36ad4652bc82f7"}
Oct 30 19:29:56.769194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.769088    4925 scope.go:122] "RemoveContainer" containerID="ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d"
Oct 30 19:29:56.775027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.774901    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659" podUID="e049d90d-2ea9-4e0a-8cca-b3261600f27e" containerName="payload-pre-processing" containerID="cri-o://b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c" gracePeriod=25
Oct 30 19:29:56.784090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.784010    4925 scope.go:122] "RemoveContainer" containerID="ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d"
Oct 30 19:29:56.784372 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:29:56.784345    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d\": container with ID starting with ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d not found: ID does not exist" containerID="ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d"
Oct 30 19:29:56.784491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.784384    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d"} err="failed to get container status \"ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d\": rpc error: code = NotFound desc = could not find container \"ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d\": container with ID starting with ce90027af07adfabea972fbe32173dcf6b038c7f2d43d618d2db27f39b13f65d not found: ID does not exist"
Oct 30 19:29:56.876837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.876700    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:56.905998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.905965    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"]
Oct 30 19:29:56.909772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.909743    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"]
Oct 30 19:29:56.917584 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:29:56.917090    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0d956_67cf_40d2_a0b0_427786ce6161.slice/crio-cfa25027fd8f5420b55eee218e4b205d08467dfd34b52751ef91496d4435e35f WatchSource:0}: Error finding container cfa25027fd8f5420b55eee218e4b205d08467dfd34b52751ef91496d4435e35f: Status 404 returned error can't find the container with id cfa25027fd8f5420b55eee218e4b205d08467dfd34b52751ef91496d4435e35f
Oct 30 19:29:56.921625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.920211    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:56.921625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.920293    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:56.921625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.920336    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:56.928657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:56.928640    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-9tntv"]
Oct 30 19:29:57.165960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.165911    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:57.231973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.231946    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:29:57.272779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.272746    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d906b5-98ed-42f0-8ea7-3ca948f57e7b" path="/var/lib/kubelet/pods/03d906b5-98ed-42f0-8ea7-3ca948f57e7b/volumes"
Oct 30 19:29:57.329470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.329363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:57.370563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.370517    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e049d90d-2ea9-4e0a-8cca-b3261600f27e-kube-api-access-pgcfh\" (UniqueName: \"kubernetes.io/projected/e049d90d-2ea9-4e0a-8cca-b3261600f27e-kube-api-access-pgcfh\") pod \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\" (UID: \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\") "
Oct 30 19:29:57.370796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.370774    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e049d90d-2ea9-4e0a-8cca-b3261600f27e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049d90d-2ea9-4e0a-8cca-b3261600f27e-plugins-config-volume\") pod \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\" (UID: \"e049d90d-2ea9-4e0a-8cca-b3261600f27e\") "
Oct 30 19:29:57.371243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.371204    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e049d90d-2ea9-4e0a-8cca-b3261600f27e-plugins-config-volume" pod "e049d90d-2ea9-4e0a-8cca-b3261600f27e" (UID: "e049d90d-2ea9-4e0a-8cca-b3261600f27e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:57.371915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.371890    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049d90d-2ea9-4e0a-8cca-b3261600f27e-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:57.373751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.373726    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e049d90d-2ea9-4e0a-8cca-b3261600f27e-kube-api-access-pgcfh" pod "e049d90d-2ea9-4e0a-8cca-b3261600f27e" (UID: "e049d90d-2ea9-4e0a-8cca-b3261600f27e"). InnerVolumeSpecName "kube-api-access-pgcfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:57.474163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.474124    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgcfh\" (UniqueName: \"kubernetes.io/projected/e049d90d-2ea9-4e0a-8cca-b3261600f27e-kube-api-access-pgcfh\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:57.665714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.659909    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:57.783109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.783071    4925 generic.go:356] "Generic (PLEG): container finished" podID="e049d90d-2ea9-4e0a-8cca-b3261600f27e" containerID="b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c" exitCode=0
Oct 30 19:29:57.785358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.783121    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"
Oct 30 19:29:57.785358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.783135    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659" event={"ID":"e049d90d-2ea9-4e0a-8cca-b3261600f27e","Type":"ContainerDied","Data":"b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c"}
Oct 30 19:29:57.785358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.783216    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659" event={"ID":"e049d90d-2ea9-4e0a-8cca-b3261600f27e","Type":"ContainerDied","Data":"b66b802802c01880c7979c0f50fda73a3286ac9fcf5b5a4a9ad04ddfa7ce056e"}
Oct 30 19:29:57.785358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.783247    4925 scope.go:122] "RemoveContainer" containerID="b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c"
Oct 30 19:29:57.792421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.792367    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" event={"ID":"b5a0d956-67cf-40d2-a0b0-427786ce6161","Type":"ContainerStarted","Data":"1cd424c0dc77ab3dda6417736cb8e6cc04aa157d640a119f13f08e2116839064"}
Oct 30 19:29:57.792568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.792466    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" event={"ID":"b5a0d956-67cf-40d2-a0b0-427786ce6161","Type":"ContainerStarted","Data":"cfa25027fd8f5420b55eee218e4b205d08467dfd34b52751ef91496d4435e35f"}
Oct 30 19:29:57.821486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.821428    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podStartSLOduration=32.821410926 podStartE2EDuration="32.821410926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:57.819968005 +0000 UTC m=+1432.722981045" watchObservedRunningTime="2026-10-30 19:29:57.821410926 +0000 UTC m=+1432.724423963"
Oct 30 19:29:57.822619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.822595    4925 scope.go:122] "RemoveContainer" containerID="b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c"
Oct 30 19:29:57.823029 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:29:57.822998    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c\": container with ID starting with b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c not found: ID does not exist" containerID="b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c"
Oct 30 19:29:57.823135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.823038    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c"} err="failed to get container status \"b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c\": rpc error: code = NotFound desc = could not find container \"b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c\": container with ID starting with b1c744e607d0f094a6fe403d5e530efb55611acbe4a949889712655332be850c not found: ID does not exist"
Oct 30 19:29:57.835813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.835788    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"]
Oct 30 19:29:57.842446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:57.842422    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ceda0-54c896b567-45659"]
Oct 30 19:29:58.545932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.545903    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:58.551649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.551622    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:58.652909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.652876    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:58.811583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.811500    4925 generic.go:356] "Generic (PLEG): container finished" podID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerID="22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458" exitCode=0
Oct 30 19:29:58.811583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.811567    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" event={"ID":"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1","Type":"ContainerDied","Data":"22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458"}
Oct 30 19:29:58.812043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.812026    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:58.812177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.812161    4925 scope.go:122] "RemoveContainer" containerID="22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458"
Oct 30 19:29:58.813955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.813934    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:29:58.814628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.814595    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:58.814767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.814705    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:58.814767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:58.814756    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:29:59.265137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:59.265102    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e049d90d-2ea9-4e0a-8cca-b3261600f27e" path="/var/lib/kubelet/pods/e049d90d-2ea9-4e0a-8cca-b3261600f27e/volumes"
Oct 30 19:29:59.424535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:59.418559    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:59.835595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:29:59.835562    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" event={"ID":"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1","Type":"ContainerStarted","Data":"414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc"}
Oct 30 19:30:00.212712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.211984    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"]
Oct 30 19:30:00.218073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.213144    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e049d90d-2ea9-4e0a-8cca-b3261600f27e" containerName="payload-pre-processing"
Oct 30 19:30:00.218073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.213170    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e049d90d-2ea9-4e0a-8cca-b3261600f27e" containerName="payload-pre-processing"
Oct 30 19:30:00.218073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.213427    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e049d90d-2ea9-4e0a-8cca-b3261600f27e" containerName="payload-pre-processing"
Oct 30 19:30:00.226710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.220994    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:00.241714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.235907    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:30:00.241714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.241495    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"]
Oct 30 19:30:00.250929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.250904    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:00.303615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.303573    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c\" (UID: \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:00.303816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.303754    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgsg\" (UniqueName: \"kubernetes.io/projected/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-kube-api-access-8zgsg\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c\" (UID: \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:00.303816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.303804    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrcp5\" (UniqueName: \"kubernetes.io/projected/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-kube-api-access-qrcp5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w\" (UID: \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:00.304052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.304026    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w\" (UID: \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:00.328073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.318738    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"]
Oct 30 19:30:00.407753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.376161    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:00.408016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.407989    4925 patch_prober.go:29] interesting pod/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:30:00.408115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.408042    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:30:00.408328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.407884    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:30:00.408659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.408628    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd487ee-ef2f-4db0-a116-7b7650babfab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-52rhp\" (UID: \"6bd487ee-ef2f-4db0-a116-7b7650babfab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:00.408786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.408711    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnqm5\" (UniqueName: \"kubernetes.io/projected/6bd487ee-ef2f-4db0-a116-7b7650babfab-kube-api-access-lnqm5\") pod \"maas-api-key-cleanup-29889810-52rhp\" (UID: \"6bd487ee-ef2f-4db0-a116-7b7650babfab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:00.456831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.410022    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w\" (UID: \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:00.456831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.425727    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w\" (UID: \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:00.456831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.426004    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c\" (UID: \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:00.456831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.426191    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgsg\" (UniqueName: \"kubernetes.io/projected/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-kube-api-access-8zgsg\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c\" (UID: \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:00.456831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.426244    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrcp5\" (UniqueName: \"kubernetes.io/projected/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-kube-api-access-qrcp5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w\" (UID: \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:00.456831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.427550    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c\" (UID: \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:00.536426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.527855    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"]
Oct 30 19:30:00.578409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.554515    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd487ee-ef2f-4db0-a116-7b7650babfab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-52rhp\" (UID: \"6bd487ee-ef2f-4db0-a116-7b7650babfab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:00.578409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.554581    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnqm5\" (UniqueName: \"kubernetes.io/projected/6bd487ee-ef2f-4db0-a116-7b7650babfab-kube-api-access-lnqm5\") pod \"maas-api-key-cleanup-29889810-52rhp\" (UID: \"6bd487ee-ef2f-4db0-a116-7b7650babfab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:00.578409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.572236    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd487ee-ef2f-4db0-a116-7b7650babfab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-52rhp\" (UID: \"6bd487ee-ef2f-4db0-a116-7b7650babfab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:00.674566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.674532    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrcp5\" (UniqueName: \"kubernetes.io/projected/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-kube-api-access-qrcp5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w\" (UID: \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:00.676919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.676895    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:00.680896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.680760    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"]
Oct 30 19:30:00.681189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.681173    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:00.681649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.681624    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgsg\" (UniqueName: \"kubernetes.io/projected/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-kube-api-access-8zgsg\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c\" (UID: \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:00.720154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.686106    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnqm5\" (UniqueName: \"kubernetes.io/projected/6bd487ee-ef2f-4db0-a116-7b7650babfab-kube-api-access-lnqm5\") pod \"maas-api-key-cleanup-29889810-52rhp\" (UID: \"6bd487ee-ef2f-4db0-a116-7b7650babfab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:00.720154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.690230    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"]
Oct 30 19:30:00.720154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.710137    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"]
Oct 30 19:30:00.730004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.721283    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f28cc08-a010-45de-9449-1e8eb1d91dfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t\" (UID: \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:00.730414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.730382    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd9jw\" (UniqueName: \"kubernetes.io/projected/5f28cc08-a010-45de-9449-1e8eb1d91dfc-kube-api-access-zd9jw\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t\" (UID: \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:00.779302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.779272    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"]
Oct 30 19:30:00.870609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.869541    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:00.871084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.871034    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f28cc08-a010-45de-9449-1e8eb1d91dfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t\" (UID: \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:00.871233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.871213    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9jw\" (UniqueName: \"kubernetes.io/projected/5f28cc08-a010-45de-9449-1e8eb1d91dfc-kube-api-access-zd9jw\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t\" (UID: \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:00.872190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.872164    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f28cc08-a010-45de-9449-1e8eb1d91dfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t\" (UID: \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:00.889402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.889375    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd9jw\" (UniqueName: \"kubernetes.io/projected/5f28cc08-a010-45de-9449-1e8eb1d91dfc-kube-api-access-zd9jw\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t\" (UID: \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:00.984381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:00.983676    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:01.127739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:01.127058    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"]
Oct 30 19:30:01.129048 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:30:01.129013    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5757f9f0_c294_4614_b5f9_c3cbb0e908e6.slice/crio-b50bbbf25c9618719da57274c8d4e440323862e44a777a17cf0fd9d81809f045 WatchSource:0}: Error finding container b50bbbf25c9618719da57274c8d4e440323862e44a777a17cf0fd9d81809f045: Status 404 returned error can't find the container with id b50bbbf25c9618719da57274c8d4e440323862e44a777a17cf0fd9d81809f045
Oct 30 19:30:01.156304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:01.156119    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:01.236777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:01.230050    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"]
Oct 30 19:30:01.331904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:01.331861    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"]
Oct 30 19:30:01.334246 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:30:01.334217    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd487ee_ef2f_4db0_a116_7b7650babfab.slice/crio-00efc1fc51305d4efedde57513f4215518d06b9160d0c60e04963b41249a1671 WatchSource:0}: Error finding container 00efc1fc51305d4efedde57513f4215518d06b9160d0c60e04963b41249a1671: Status 404 returned error can't find the container with id 00efc1fc51305d4efedde57513f4215518d06b9160d0c60e04963b41249a1671
Oct 30 19:30:01.419405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:01.415506    4925 patch_prober.go:29] interesting pod/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:30:01.419405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:01.415561    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:30:01.490489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:01.490462    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"]
Oct 30 19:30:01.492459 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:30:01.492433    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f28cc08_a010_45de_9449_1e8eb1d91dfc.slice/crio-466505fd159fc2ac3001e18571a942e150f1b7f5c4a0ebb5e28fdbc1cd3a01e7 WatchSource:0}: Error finding container 466505fd159fc2ac3001e18571a942e150f1b7f5c4a0ebb5e28fdbc1cd3a01e7: Status 404 returned error can't find the container with id 466505fd159fc2ac3001e18571a942e150f1b7f5c4a0ebb5e28fdbc1cd3a01e7
Oct 30 19:30:02.079712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.076151    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" event={"ID":"5f28cc08-a010-45de-9449-1e8eb1d91dfc","Type":"ContainerStarted","Data":"466505fd159fc2ac3001e18571a942e150f1b7f5c4a0ebb5e28fdbc1cd3a01e7"}
Oct 30 19:30:02.099345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.099191    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:02.103421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.101508    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" event={"ID":"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949","Type":"ContainerStarted","Data":"62dbfb6c9e262673a52ffeaf5ee687fa1aa6ffd54c540148f1677fe55a2727ae"}
Oct 30 19:30:02.103421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.101575    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" event={"ID":"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949","Type":"ContainerStarted","Data":"27dcdb11fd5f312d093b5e303e7685aa38b42627f61c034b6b3e67bf0738b233"}
Oct 30 19:30:02.109647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.109611    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" event={"ID":"5757f9f0-c294-4614-b5f9-c3cbb0e908e6","Type":"ContainerStarted","Data":"72a2ccce733f8486dc0d7a8c70e1ef7a4dbffdb7be1ffa64615d9655d7f8d505"}
Oct 30 19:30:02.110099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.110052    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" event={"ID":"5757f9f0-c294-4614-b5f9-c3cbb0e908e6","Type":"ContainerStarted","Data":"b50bbbf25c9618719da57274c8d4e440323862e44a777a17cf0fd9d81809f045"}
Oct 30 19:30:02.209940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.192381    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" event={"ID":"6bd487ee-ef2f-4db0-a116-7b7650babfab","Type":"ContainerStarted","Data":"00efc1fc51305d4efedde57513f4215518d06b9160d0c60e04963b41249a1671"}
Oct 30 19:30:02.210506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.210445    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" podStartSLOduration=2.210423725 podStartE2EDuration="2.210423725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.124856434 +0000 UTC m=+1437.027869469" watchObservedRunningTime="2026-10-30 19:30:02.210423725 +0000 UTC m=+1437.113436765"
Oct 30 19:30:02.242716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.241647    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" podStartSLOduration=2.241625907 podStartE2EDuration="2.241625907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.220193044 +0000 UTC m=+1437.123206082" watchObservedRunningTime="2026-10-30 19:30:02.241625907 +0000 UTC m=+1437.144638945"
Oct 30 19:30:02.322795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.322721    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:02.408522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.408065    4925 patch_prober.go:29] interesting pod/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:30:02.408522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.408137    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:30:02.542893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:02.542802    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:03.227643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.227608    4925 generic.go:356] "Generic (PLEG): container finished" podID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerID="c113483854ca363cf3f26bfcd573410886da39d9fac9e6653add389243593bfa" exitCode=60
Oct 30 19:30:03.228154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.227652    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" event={"ID":"5f28cc08-a010-45de-9449-1e8eb1d91dfc","Type":"ContainerDied","Data":"c113483854ca363cf3f26bfcd573410886da39d9fac9e6653add389243593bfa"}
Oct 30 19:30:03.228154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.228071    4925 scope.go:122] "RemoveContainer" containerID="c113483854ca363cf3f26bfcd573410886da39d9fac9e6653add389243593bfa"
Oct 30 19:30:03.229652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.229631    4925 generic.go:356] "Generic (PLEG): container finished" podID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerID="62dbfb6c9e262673a52ffeaf5ee687fa1aa6ffd54c540148f1677fe55a2727ae" exitCode=60
Oct 30 19:30:03.229769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.229721    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" event={"ID":"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949","Type":"ContainerDied","Data":"62dbfb6c9e262673a52ffeaf5ee687fa1aa6ffd54c540148f1677fe55a2727ae"}
Oct 30 19:30:03.230100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.230015    4925 scope.go:122] "RemoveContainer" containerID="62dbfb6c9e262673a52ffeaf5ee687fa1aa6ffd54c540148f1677fe55a2727ae"
Oct 30 19:30:03.231637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.231607    4925 generic.go:356] "Generic (PLEG): container finished" podID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerID="72a2ccce733f8486dc0d7a8c70e1ef7a4dbffdb7be1ffa64615d9655d7f8d505" exitCode=60
Oct 30 19:30:03.231792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.231682    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" event={"ID":"5757f9f0-c294-4614-b5f9-c3cbb0e908e6","Type":"ContainerDied","Data":"72a2ccce733f8486dc0d7a8c70e1ef7a4dbffdb7be1ffa64615d9655d7f8d505"}
Oct 30 19:30:03.232006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.231988    4925 scope.go:122] "RemoveContainer" containerID="72a2ccce733f8486dc0d7a8c70e1ef7a4dbffdb7be1ffa64615d9655d7f8d505"
Oct 30 19:30:03.234023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.233990    4925 generic.go:356] "Generic (PLEG): container finished" podID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerID="d4df11f6075b99634adc093f4d1b248b000f33a97ddd2b64588015789ae5a350" exitCode=60
Oct 30 19:30:03.234128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.234068    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" event={"ID":"6bd487ee-ef2f-4db0-a116-7b7650babfab","Type":"ContainerDied","Data":"d4df11f6075b99634adc093f4d1b248b000f33a97ddd2b64588015789ae5a350"}
Oct 30 19:30:03.234325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.234308    4925 scope.go:122] "RemoveContainer" containerID="d4df11f6075b99634adc093f4d1b248b000f33a97ddd2b64588015789ae5a350"
Oct 30 19:30:03.423886 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.423838    4925 patch_prober.go:29] interesting pod/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:30:03.424073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.423901    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:30:03.824980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:03.818324    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:04.246218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:04.246185    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" event={"ID":"5f28cc08-a010-45de-9449-1e8eb1d91dfc","Type":"ContainerStarted","Data":"75c168dec6597f87fda3b68ce0c7852a62623ad0671762b498a147c7512089fe"}
Oct 30 19:30:04.248260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:04.248236    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" event={"ID":"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949","Type":"ContainerStarted","Data":"7b2ee21b7cdf9ea9e765588ffa8ea39006ce31bb7e6e000c5b2aa06202d1052c"}
Oct 30 19:30:04.250823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:04.250799    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" event={"ID":"5757f9f0-c294-4614-b5f9-c3cbb0e908e6","Type":"ContainerStarted","Data":"8d0f62ce0aff8570f771bfbeabb41b6061fd78bd040df3b8f0827e3dbcaebbf0"}
Oct 30 19:30:04.252592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:04.252570    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" event={"ID":"6bd487ee-ef2f-4db0-a116-7b7650babfab","Type":"ContainerStarted","Data":"24763723da445c603fea502b429f0ea4f4ef77ab2b413f2df511cf1734f2060e"}
Oct 30 19:30:04.279278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:04.279228    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" podStartSLOduration=4.279214201 podStartE2EDuration="4.279214201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:04.274752954 +0000 UTC m=+1439.177765991" watchObservedRunningTime="2026-10-30 19:30:04.279214201 +0000 UTC m=+1439.182227265"
Oct 30 19:30:04.327299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:04.327232    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" podStartSLOduration=4.327212382 podStartE2EDuration="4.327212382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:04.314111053 +0000 UTC m=+1439.217124091" watchObservedRunningTime="2026-10-30 19:30:04.327212382 +0000 UTC m=+1439.230225420"
Oct 30 19:30:04.422784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:04.418123    4925 patch_prober.go:29] interesting pod/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:30:04.422784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:04.418186    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:30:05.041765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.041729    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:05.144025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.143995    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:30:05.265983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.265953    4925 generic.go:356] "Generic (PLEG): container finished" podID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerID="75c168dec6597f87fda3b68ce0c7852a62623ad0671762b498a147c7512089fe" exitCode=60
Oct 30 19:30:05.266453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.266303    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" event={"ID":"5f28cc08-a010-45de-9449-1e8eb1d91dfc","Type":"ContainerDied","Data":"75c168dec6597f87fda3b68ce0c7852a62623ad0671762b498a147c7512089fe"}
Oct 30 19:30:05.266453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.266372    4925 scope.go:122] "RemoveContainer" containerID="c113483854ca363cf3f26bfcd573410886da39d9fac9e6653add389243593bfa"
Oct 30 19:30:05.266580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.266486    4925 scope.go:122] "RemoveContainer" containerID="75c168dec6597f87fda3b68ce0c7852a62623ad0671762b498a147c7512089fe"
Oct 30 19:30:05.267002 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:05.266975    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t_odh-ai-gateway-infra(5f28cc08-a010-45de-9449-1e8eb1d91dfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc"
Oct 30 19:30:05.268257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.268235    4925 generic.go:356] "Generic (PLEG): container finished" podID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerID="7b2ee21b7cdf9ea9e765588ffa8ea39006ce31bb7e6e000c5b2aa06202d1052c" exitCode=60
Oct 30 19:30:05.268357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.268323    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" event={"ID":"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949","Type":"ContainerDied","Data":"7b2ee21b7cdf9ea9e765588ffa8ea39006ce31bb7e6e000c5b2aa06202d1052c"}
Oct 30 19:30:05.268658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.268642    4925 scope.go:122] "RemoveContainer" containerID="7b2ee21b7cdf9ea9e765588ffa8ea39006ce31bb7e6e000c5b2aa06202d1052c"
Oct 30 19:30:05.269068 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:05.269047    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c_odh-ai-gateway-infra(1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949"
Oct 30 19:30:05.270349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.270330    4925 generic.go:356] "Generic (PLEG): container finished" podID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerID="8d0f62ce0aff8570f771bfbeabb41b6061fd78bd040df3b8f0827e3dbcaebbf0" exitCode=60
Oct 30 19:30:05.270443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.270402    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" event={"ID":"5757f9f0-c294-4614-b5f9-c3cbb0e908e6","Type":"ContainerDied","Data":"8d0f62ce0aff8570f771bfbeabb41b6061fd78bd040df3b8f0827e3dbcaebbf0"}
Oct 30 19:30:05.270747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.270728    4925 scope.go:122] "RemoveContainer" containerID="8d0f62ce0aff8570f771bfbeabb41b6061fd78bd040df3b8f0827e3dbcaebbf0"
Oct 30 19:30:05.271098 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:05.271077    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w_odh-ai-gateway-infra(5757f9f0-c294-4614-b5f9-c3cbb0e908e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6"
Oct 30 19:30:05.272529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.272483    4925 generic.go:356] "Generic (PLEG): container finished" podID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerID="24763723da445c603fea502b429f0ea4f4ef77ab2b413f2df511cf1734f2060e" exitCode=60
Oct 30 19:30:05.272529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.272524    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" event={"ID":"6bd487ee-ef2f-4db0-a116-7b7650babfab","Type":"ContainerDied","Data":"24763723da445c603fea502b429f0ea4f4ef77ab2b413f2df511cf1734f2060e"}
Oct 30 19:30:05.272821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.272805    4925 scope.go:122] "RemoveContainer" containerID="24763723da445c603fea502b429f0ea4f4ef77ab2b413f2df511cf1734f2060e"
Oct 30 19:30:05.273098 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:05.273082    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-52rhp_odh-ai-gateway-infra(6bd487ee-ef2f-4db0-a116-7b7650babfab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab"
Oct 30 19:30:05.281888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.281865    4925 scope.go:122] "RemoveContainer" containerID="62dbfb6c9e262673a52ffeaf5ee687fa1aa6ffd54c540148f1677fe55a2727ae"
Oct 30 19:30:05.316285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.316207    4925 scope.go:122] "RemoveContainer" containerID="72a2ccce733f8486dc0d7a8c70e1ef7a4dbffdb7be1ffa64615d9655d7f8d505"
Oct 30 19:30:05.320465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.320255    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:05.343846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.343824    4925 scope.go:122] "RemoveContainer" containerID="d4df11f6075b99634adc093f4d1b248b000f33a97ddd2b64588015789ae5a350"
Oct 30 19:30:05.412713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.408055    4925 patch_prober.go:29] interesting pod/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:30:05.412713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.408112    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:30:05.580311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:05.580214    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:06.042711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.041980    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"]
Oct 30 19:30:06.042711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.042245    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy" containerID="cri-o://c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b" gracePeriod=30
Oct 30 19:30:06.047381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.047354    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"]
Oct 30 19:30:06.047583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.047550    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy" containerID="cri-o://e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f" gracePeriod=30
Oct 30 19:30:06.056466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.056436    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"]
Oct 30 19:30:06.056739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.056672    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy" containerID="cri-o://720b5bda5b6a78b8f611061dfc2fa514195ee10ec322e6071d93cc4f0e94c6c9" gracePeriod=30
Oct 30 19:30:06.284208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.284179    4925 scope.go:122] "RemoveContainer" containerID="75c168dec6597f87fda3b68ce0c7852a62623ad0671762b498a147c7512089fe"
Oct 30 19:30:06.284656 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:06.284631    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t_odh-ai-gateway-infra(5f28cc08-a010-45de-9449-1e8eb1d91dfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc"
Oct 30 19:30:06.285947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.285923    4925 scope.go:122] "RemoveContainer" containerID="7b2ee21b7cdf9ea9e765588ffa8ea39006ce31bb7e6e000c5b2aa06202d1052c"
Oct 30 19:30:06.286297 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:06.286267    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c_odh-ai-gateway-infra(1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949"
Oct 30 19:30:06.287593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.287575    4925 scope.go:122] "RemoveContainer" containerID="8d0f62ce0aff8570f771bfbeabb41b6061fd78bd040df3b8f0827e3dbcaebbf0"
Oct 30 19:30:06.287948 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:06.287898    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w_odh-ai-gateway-infra(5757f9f0-c294-4614-b5f9-c3cbb0e908e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6"
Oct 30 19:30:06.289187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.289170    4925 scope.go:122] "RemoveContainer" containerID="24763723da445c603fea502b429f0ea4f4ef77ab2b413f2df511cf1734f2060e"
Oct 30 19:30:06.289401 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:06.289384    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-52rhp_odh-ai-gateway-infra(6bd487ee-ef2f-4db0-a116-7b7650babfab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab"
Oct 30 19:30:06.413617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.413586    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:30:06.414927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:06.414906    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:30:07.106211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.106180    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:07.139074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.139045    4925 scope.go:122] "RemoveContainer" containerID="e5fceb67450ffbd358e2641df6f85cd965662968868b7a8287c5c4f566ae3570"
Oct 30 19:30:07.150415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.150383    4925 scope.go:122] "RemoveContainer" containerID="e99f429e99db5050791145deee8280f6621b26c9e200afefc49f5452602d91fc"
Oct 30 19:30:07.161475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.161447    4925 scope.go:122] "RemoveContainer" containerID="c61f3b0c29511efb750a3fab667a7cd99dd9e1ec96bc77452cc4d6d1e6fa7673"
Oct 30 19:30:07.171866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.171843    4925 scope.go:122] "RemoveContainer" containerID="7220a3a96a113ebf9267af728a1c195d05e20cefd3026b814fd1c084b5523800"
Oct 30 19:30:07.183485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.183459    4925 scope.go:122] "RemoveContainer" containerID="9dc4b6a8acd8d592c9924989214f6136398cc8c7aa5ab7b29f81d2cc97de26a6"
Oct 30 19:30:07.194720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.194674    4925 scope.go:122] "RemoveContainer" containerID="496757f23390fcca8b0b9a00706bab1eb9f867364cfc42c3c73e91dfb5e35adf"
Oct 30 19:30:07.210262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.210237    4925 scope.go:122] "RemoveContainer" containerID="2ee3f4866788745a6e3a5ab9534537e690aee5d0981672545198bab08df8921f"
Oct 30 19:30:07.221357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.221334    4925 scope.go:122] "RemoveContainer" containerID="e659d343dfc752551cbf2dc3f490e2528774507cc719e6b54fbe4637c36a2982"
Oct 30 19:30:07.231952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.231930    4925 scope.go:122] "RemoveContainer" containerID="14d8e9dcbcea3b6481ef3306997761b4d93d104e6f7c9d7e00975e0f5afc2fad"
Oct 30 19:30:07.371770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.367239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:07.910307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:07.910206    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:08.468059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:08.468027    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:09.491045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:09.491009    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:10.360161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:10.359750    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:11.133931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.133897    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:11.323459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.323325    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:11.455370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.455340    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:30:11.500295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.500265    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:30:11.528933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.528903    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"]
Oct 30 19:30:11.530446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.530420    4925 cpu_manager.go:414] "containerMap: removing container" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy"
Oct 30 19:30:11.530628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.530612    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy"
Oct 30 19:30:11.530810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.530796    4925 cpu_manager.go:414] "containerMap: removing container" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy"
Oct 30 19:30:11.530910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.530898    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy"
Oct 30 19:30:11.531189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.531173    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerName="istio-proxy"
Oct 30 19:30:11.531338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.531326    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerName="istio-proxy"
Oct 30 19:30:11.536276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.536260    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:11.543624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.543596    4925 generic.go:356] "Generic (PLEG): container finished" podID="61a9e3ec-6b63-4906-8647-91384664f2f9" containerID="c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b" exitCode=0
Oct 30 19:30:11.544126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.544098    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" event={"ID":"61a9e3ec-6b63-4906-8647-91384664f2f9","Type":"ContainerDied","Data":"c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b"}
Oct 30 19:30:11.544209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.544155    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"
Oct 30 19:30:11.544269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.544208    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs" event={"ID":"61a9e3ec-6b63-4906-8647-91384664f2f9","Type":"ContainerDied","Data":"2518c37170991f1ae10a0af02b653c0e726da61d893d2e5d751e2c08870c0e85"}
Oct 30 19:30:11.544269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.544247    4925 scope.go:122] "RemoveContainer" containerID="c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b"
Oct 30 19:30:11.546533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.546512    4925 generic.go:356] "Generic (PLEG): container finished" podID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" containerID="e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f" exitCode=0
Oct 30 19:30:11.546659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.546635    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"
Oct 30 19:30:11.546844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.546826    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" event={"ID":"56b37d28-6688-46cb-b2e4-b1bc7495eaa0","Type":"ContainerDied","Data":"e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f"}
Oct 30 19:30:11.550094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.550066    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb" event={"ID":"56b37d28-6688-46cb-b2e4-b1bc7495eaa0","Type":"ContainerDied","Data":"c5d435cd90dca621f42a015e5f32f339df7f3bad8d89fe8f06003377e16c3c21"}
Oct 30 19:30:11.550236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.549876    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-socket" pod "61a9e3ec-6b63-4906-8647-91384664f2f9" (UID: "61a9e3ec-6b63-4906-8647-91384664f2f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.550407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.549138    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-socket\") pod \"61a9e3ec-6b63-4906-8647-91384664f2f9\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") "
Oct 30 19:30:11.550857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.550839    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a9e3ec-6b63-4906-8647-91384664f2f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a9e3ec-6b63-4906-8647-91384664f2f9-istiod-ca-cert\") pod \"61a9e3ec-6b63-4906-8647-91384664f2f9\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") "
Oct 30 19:30:11.552478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.552460    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-kube-api-access-z5qkt\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-kube-api-access-z5qkt\") pod \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") "
Oct 30 19:30:11.552675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.552661    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-socket\") pod \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553107    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-envoy\") pod \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.552394    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a9e3ec-6b63-4906-8647-91384664f2f9-istiod-ca-cert" pod "61a9e3ec-6b63-4906-8647-91384664f2f9" (UID: "61a9e3ec-6b63-4906-8647-91384664f2f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553042    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-socket" pod "56b37d28-6688-46cb-b2e4-b1bc7495eaa0" (UID: "56b37d28-6688-46cb-b2e4-b1bc7495eaa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553162    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istiod-ca-cert\") pod \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.552469    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"]
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553191    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-data\") pod \"61a9e3ec-6b63-4906-8647-91384664f2f9\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553274    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-certs\") pod \"61a9e3ec-6b63-4906-8647-91384664f2f9\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553314    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-certs\") pod \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553351    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-podinfo\") pod \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553417    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-podinfo\") pod \"61a9e3ec-6b63-4906-8647-91384664f2f9\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553463    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-credential-socket\") pod \"61a9e3ec-6b63-4906-8647-91384664f2f9\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553536    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istiod-ca-cert" pod "56b37d28-6688-46cb-b2e4-b1bc7495eaa0" (UID: "56b37d28-6688-46cb-b2e4-b1bc7495eaa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553564    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-token\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-token\") pod \"61a9e3ec-6b63-4906-8647-91384664f2f9\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553667    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-data\") pod \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553766    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-envoy\") pod \"61a9e3ec-6b63-4906-8647-91384664f2f9\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553826    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-kube-api-access-rx2vq\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-kube-api-access-rx2vq\") pod \"61a9e3ec-6b63-4906-8647-91384664f2f9\" (UID: \"61a9e3ec-6b63-4906-8647-91384664f2f9\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553838    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-certs" pod "56b37d28-6688-46cb-b2e4-b1bc7495eaa0" (UID: "56b37d28-6688-46cb-b2e4-b1bc7495eaa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553859    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-credential-socket\") pod \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.553947    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-token\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-token\") pod \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\" (UID: \"56b37d28-6688-46cb-b2e4-b1bc7495eaa0\") "
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.554024    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-data" pod "61a9e3ec-6b63-4906-8647-91384664f2f9" (UID: "61a9e3ec-6b63-4906-8647-91384664f2f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.554473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.554208    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-certs" pod "61a9e3ec-6b63-4906-8647-91384664f2f9" (UID: "61a9e3ec-6b63-4906-8647-91384664f2f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.555923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.555334    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a9e3ec-6b63-4906-8647-91384664f2f9-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.555923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.555357    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.555923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.555372    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.555923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.555387    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.555923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.555401    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.555923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.555416    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.555923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.555430    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.556302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.556053    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-kube-api-access-z5qkt" pod "56b37d28-6688-46cb-b2e4-b1bc7495eaa0" (UID: "56b37d28-6688-46cb-b2e4-b1bc7495eaa0"). InnerVolumeSpecName "kube-api-access-z5qkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:11.556302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.556251    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-02feaa\""
Oct 30 19:30:11.556838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.556817    4925 generic.go:356] "Generic (PLEG): container finished" podID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerID="720b5bda5b6a78b8f611061dfc2fa514195ee10ec322e6071d93cc4f0e94c6c9" exitCode=0
Oct 30 19:30:11.556988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.556971    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" event={"ID":"306e5049-a83d-4e23-bde7-66c10dfe4a63","Type":"ContainerDied","Data":"720b5bda5b6a78b8f611061dfc2fa514195ee10ec322e6071d93cc4f0e94c6c9"}
Oct 30 19:30:11.557082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.557057    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-envoy" pod "56b37d28-6688-46cb-b2e4-b1bc7495eaa0" (UID: "56b37d28-6688-46cb-b2e4-b1bc7495eaa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.558921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.558897    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-02feaa-dockercfg-qrbgf\""
Oct 30 19:30:11.559579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.559554    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-token" pod "56b37d28-6688-46cb-b2e4-b1bc7495eaa0" (UID: "56b37d28-6688-46cb-b2e4-b1bc7495eaa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:11.563163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.563134    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-podinfo" pod "56b37d28-6688-46cb-b2e4-b1bc7495eaa0" (UID: "56b37d28-6688-46cb-b2e4-b1bc7495eaa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:11.565138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.565023    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-credential-socket" pod "61a9e3ec-6b63-4906-8647-91384664f2f9" (UID: "61a9e3ec-6b63-4906-8647-91384664f2f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.565492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.565216    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-data" pod "56b37d28-6688-46cb-b2e4-b1bc7495eaa0" (UID: "56b37d28-6688-46cb-b2e4-b1bc7495eaa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.565661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.565637    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-credential-socket" pod "56b37d28-6688-46cb-b2e4-b1bc7495eaa0" (UID: "56b37d28-6688-46cb-b2e4-b1bc7495eaa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.565900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.565881    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-kube-api-access-rx2vq" pod "61a9e3ec-6b63-4906-8647-91384664f2f9" (UID: "61a9e3ec-6b63-4906-8647-91384664f2f9"). InnerVolumeSpecName "kube-api-access-rx2vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:11.567135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.567112    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-envoy" pod "61a9e3ec-6b63-4906-8647-91384664f2f9" (UID: "61a9e3ec-6b63-4906-8647-91384664f2f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.567226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.567160    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-token" pod "61a9e3ec-6b63-4906-8647-91384664f2f9" (UID: "61a9e3ec-6b63-4906-8647-91384664f2f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:11.577161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.571978    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-podinfo" pod "61a9e3ec-6b63-4906-8647-91384664f2f9" (UID: "61a9e3ec-6b63-4906-8647-91384664f2f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:11.577161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.572208    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:30:11.577161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.572342    4925 scope.go:122] "RemoveContainer" containerID="c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b"
Oct 30 19:30:11.577161 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:11.573151    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b\": container with ID starting with c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b not found: ID does not exist" containerID="c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b"
Oct 30 19:30:11.577161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.573436    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b"} err="failed to get container status \"c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b\": rpc error: code = NotFound desc = could not find container \"c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b\": container with ID starting with c0b2c6bcb181388814fa4c1659d9fc48ec6efa65d88d7fb54a94443e17642c8b not found: ID does not exist"
Oct 30 19:30:11.577161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.573477    4925 scope.go:122] "RemoveContainer" containerID="e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f"
Oct 30 19:30:11.577161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.576374    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"]
Oct 30 19:30:11.577882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.577859    4925 cpu_manager.go:414] "containerMap: removing container" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy"
Oct 30 19:30:11.577882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.577882    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy"
Oct 30 19:30:11.578263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.578245    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" containerName="istio-proxy"
Oct 30 19:30:11.592329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.592301    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.619823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.619786    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"]
Oct 30 19:30:11.622130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.622105    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:11.626164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.626144    4925 scope.go:122] "RemoveContainer" containerID="e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f"
Oct 30 19:30:11.626585 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:11.626561    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f\": container with ID starting with e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f not found: ID does not exist" containerID="e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f"
Oct 30 19:30:11.626669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.626598    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f"} err="failed to get container status \"e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f\": rpc error: code = NotFound desc = could not find container \"e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f\": container with ID starting with e30f973891ab32887a8a34c4e4257e4ba7f5182d15573e5bb3e8de102b7f476f not found: ID does not exist"
Oct 30 19:30:11.652137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.652110    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8e6159-serving-cert\""
Oct 30 19:30:11.656848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.656817    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-envoy\") pod \"306e5049-a83d-4e23-bde7-66c10dfe4a63\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") "
Oct 30 19:30:11.656993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.656915    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-token\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-token\") pod \"306e5049-a83d-4e23-bde7-66c10dfe4a63\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") "
Oct 30 19:30:11.656993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.656955    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-kube-api-access-6cs9x\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-kube-api-access-6cs9x\") pod \"306e5049-a83d-4e23-bde7-66c10dfe4a63\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") "
Oct 30 19:30:11.657131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.657106    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-socket\") pod \"306e5049-a83d-4e23-bde7-66c10dfe4a63\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") "
Oct 30 19:30:11.657186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.657134    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-data\") pod \"306e5049-a83d-4e23-bde7-66c10dfe4a63\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") "
Oct 30 19:30:11.657308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.657290    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306e5049-a83d-4e23-bde7-66c10dfe4a63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306e5049-a83d-4e23-bde7-66c10dfe4a63-istiod-ca-cert\") pod \"306e5049-a83d-4e23-bde7-66c10dfe4a63\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") "
Oct 30 19:30:11.657415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.657399    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-podinfo\") pod \"306e5049-a83d-4e23-bde7-66c10dfe4a63\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") "
Oct 30 19:30:11.657481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.657442    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-certs\") pod \"306e5049-a83d-4e23-bde7-66c10dfe4a63\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") "
Oct 30 19:30:11.657542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.657481    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-credential-socket\") pod \"306e5049-a83d-4e23-bde7-66c10dfe4a63\" (UID: \"306e5049-a83d-4e23-bde7-66c10dfe4a63\") "
Oct 30 19:30:11.657665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.657640    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-data" pod "306e5049-a83d-4e23-bde7-66c10dfe4a63" (UID: "306e5049-a83d-4e23-bde7-66c10dfe4a63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.657875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.657852    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-socket" pod "306e5049-a83d-4e23-bde7-66c10dfe4a63" (UID: "306e5049-a83d-4e23-bde7-66c10dfe4a63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.657951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.657886    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvhd\" (UniqueName: \"kubernetes.io/projected/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-kube-api-access-plvhd\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.658084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658065    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-metrics-certs\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.658150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658104    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.658209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658175    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llk6x\" (UniqueName: \"kubernetes.io/projected/f08e19cf-765f-46fc-997c-c25e6a7dae3a-kube-api-access-llk6x\") pod \"payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh\" (UID: \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:11.658263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658251    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08e19cf-765f-46fc-997c-c25e6a7dae3a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh\" (UID: \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:11.658357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658340    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-maas-api-tls\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.658572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658555    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.658651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658578    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.658651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658593    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.658651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658609    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx2vq\" (UniqueName: \"kubernetes.io/projected/61a9e3ec-6b63-4906-8647-91384664f2f9-kube-api-access-rx2vq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.658651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658625    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.658651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658638    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.658651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658651    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.659058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658664    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.659058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658678    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5qkt\" (UniqueName: \"kubernetes.io/projected/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-kube-api-access-z5qkt\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.659058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658747    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.659058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658761    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b37d28-6688-46cb-b2e4-b1bc7495eaa0-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.659058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658774    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a9e3ec-6b63-4906-8647-91384664f2f9-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.659058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.658787    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a9e3ec-6b63-4906-8647-91384664f2f9-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.659058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.659029    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-certs" pod "306e5049-a83d-4e23-bde7-66c10dfe4a63" (UID: "306e5049-a83d-4e23-bde7-66c10dfe4a63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.659403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.659082    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306e5049-a83d-4e23-bde7-66c10dfe4a63-istiod-ca-cert" pod "306e5049-a83d-4e23-bde7-66c10dfe4a63" (UID: "306e5049-a83d-4e23-bde7-66c10dfe4a63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:11.659403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.659220    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-credential-socket" pod "306e5049-a83d-4e23-bde7-66c10dfe4a63" (UID: "306e5049-a83d-4e23-bde7-66c10dfe4a63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.660271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.660244    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-envoy" pod "306e5049-a83d-4e23-bde7-66c10dfe4a63" (UID: "306e5049-a83d-4e23-bde7-66c10dfe4a63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:11.661995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.661969    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-kube-api-access-6cs9x" pod "306e5049-a83d-4e23-bde7-66c10dfe4a63" (UID: "306e5049-a83d-4e23-bde7-66c10dfe4a63"). InnerVolumeSpecName "kube-api-access-6cs9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:11.662145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.662122    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-podinfo" pod "306e5049-a83d-4e23-bde7-66c10dfe4a63" (UID: "306e5049-a83d-4e23-bde7-66c10dfe4a63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:11.662296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.662258    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-token" pod "306e5049-a83d-4e23-bde7-66c10dfe4a63" (UID: "306e5049-a83d-4e23-bde7-66c10dfe4a63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:11.762582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.762542    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-maas-api-tls\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.762866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.762835    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvhd\" (UniqueName: \"kubernetes.io/projected/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-kube-api-access-plvhd\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.763020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.762997    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-metrics-certs\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.763345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763077    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.763345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763182    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llk6x\" (UniqueName: \"kubernetes.io/projected/f08e19cf-765f-46fc-997c-c25e6a7dae3a-kube-api-access-llk6x\") pod \"payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh\" (UID: \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:11.763345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763258    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08e19cf-765f-46fc-997c-c25e6a7dae3a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh\" (UID: \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:11.763564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763408    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.763564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763426    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.763564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763440    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cs9x\" (UniqueName: \"kubernetes.io/projected/306e5049-a83d-4e23-bde7-66c10dfe4a63-kube-api-access-6cs9x\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.763564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763454    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306e5049-a83d-4e23-bde7-66c10dfe4a63-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.763564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763467    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306e5049-a83d-4e23-bde7-66c10dfe4a63-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.763564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763481    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.763564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.763494    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306e5049-a83d-4e23-bde7-66c10dfe4a63-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.764519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.764494    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08e19cf-765f-46fc-997c-c25e6a7dae3a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh\" (UID: \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:11.764650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.764598    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.767150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.767128    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-maas-api-tls\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.767892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.767846    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-metrics-certs\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.774766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.774747    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvhd\" (UniqueName: \"kubernetes.io/projected/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-kube-api-access-plvhd\") pod \"maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.775833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.775810    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llk6x\" (UniqueName: \"kubernetes.io/projected/f08e19cf-765f-46fc-997c-c25e6a7dae3a-kube-api-access-llk6x\") pod \"payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh\" (UID: \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:11.866921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.866889    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:11.875942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.875914    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"]
Oct 30 19:30:11.883422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.883390    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-98fjs"]
Oct 30 19:30:11.905390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.905345    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"]
Oct 30 19:30:11.918187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.918149    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:11.920762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:11.920734    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-28wrb"]
Oct 30 19:30:12.181795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.181767    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"]
Oct 30 19:30:12.182404 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:30:12.182377    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08e19cf_765f_46fc_997c_c25e6a7dae3a.slice/crio-571cf12bea8c2ed6a195931fde554c762dc1840a2dfcfe2e4d430fdb110a05bf WatchSource:0}: Error finding container 571cf12bea8c2ed6a195931fde554c762dc1840a2dfcfe2e4d430fdb110a05bf: Status 404 returned error can't find the container with id 571cf12bea8c2ed6a195931fde554c762dc1840a2dfcfe2e4d430fdb110a05bf
Oct 30 19:30:12.251470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.251441    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"]
Oct 30 19:30:12.252136 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:30:12.252112    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf332f62_5e55_4fc7_8e5b_71fe1129d6c1.slice/crio-f7e4a3cc7cba696a37c4fb051f970ccfd0812d410a93d84215b7743e4825ea9a WatchSource:0}: Error finding container f7e4a3cc7cba696a37c4fb051f970ccfd0812d410a93d84215b7743e4825ea9a: Status 404 returned error can't find the container with id f7e4a3cc7cba696a37c4fb051f970ccfd0812d410a93d84215b7743e4825ea9a
Oct 30 19:30:12.253907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.253888    4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:30:12.386465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.386405    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:12.568548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.568451    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh" event={"ID":"f08e19cf-765f-46fc-997c-c25e6a7dae3a","Type":"ContainerStarted","Data":"1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1"}
Oct 30 19:30:12.568548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.568512    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh" event={"ID":"f08e19cf-765f-46fc-997c-c25e6a7dae3a","Type":"ContainerStarted","Data":"571cf12bea8c2ed6a195931fde554c762dc1840a2dfcfe2e4d430fdb110a05bf"}
Oct 30 19:30:12.569561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.569536    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:12.574197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.574173    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"
Oct 30 19:30:12.574335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.574199    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr" event={"ID":"306e5049-a83d-4e23-bde7-66c10dfe4a63","Type":"ContainerDied","Data":"392fced859e8f3fe87c01d16430a524458e8899690f5498546d23aaf223ddd64"}
Oct 30 19:30:12.574335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.574266    4925 scope.go:122] "RemoveContainer" containerID="720b5bda5b6a78b8f611061dfc2fa514195ee10ec322e6071d93cc4f0e94c6c9"
Oct 30 19:30:12.577528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.577501    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn" event={"ID":"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1","Type":"ContainerStarted","Data":"f7e4a3cc7cba696a37c4fb051f970ccfd0812d410a93d84215b7743e4825ea9a"}
Oct 30 19:30:12.593075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.593023    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh" podStartSLOduration=5.593005397 podStartE2EDuration="5.593005397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:12.586902328 +0000 UTC m=+1447.489915362" watchObservedRunningTime="2026-10-30 19:30:12.593005397 +0000 UTC m=+1447.496018438"
Oct 30 19:30:12.615609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.615580    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"]
Oct 30 19:30:12.623660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:12.623626    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-88jvr"]
Oct 30 19:30:13.261045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:13.261004    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:13.271653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:13.271619    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306e5049-a83d-4e23-bde7-66c10dfe4a63" path="/var/lib/kubelet/pods/306e5049-a83d-4e23-bde7-66c10dfe4a63/volumes"
Oct 30 19:30:13.272269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:13.272248    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b37d28-6688-46cb-b2e4-b1bc7495eaa0" path="/var/lib/kubelet/pods/56b37d28-6688-46cb-b2e4-b1bc7495eaa0/volumes"
Oct 30 19:30:13.273014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:13.272987    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a9e3ec-6b63-4906-8647-91384664f2f9" path="/var/lib/kubelet/pods/61a9e3ec-6b63-4906-8647-91384664f2f9/volumes"
Oct 30 19:30:13.588784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:13.588678    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn" event={"ID":"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1","Type":"ContainerStarted","Data":"df4dafa7b8310dfd9603114343bc15b64b22d2d08d4ea67e86a2140ac3c2fac2"}
Oct 30 19:30:13.588947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:13.588777    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:13.610941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:13.610898    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn" podStartSLOduration=46.118330547 podStartE2EDuration="46.610884039s" totalImagesPullingTime="492.553492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:13.607660953 +0000 UTC m=+1448.510673990" watchObservedRunningTime="2026-10-30 19:30:13.610884039 +0000 UTC m=+1448.513897075"
Oct 30 19:30:14.282066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:14.282025    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:14.781120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:14.781088    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:14.951347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:14.951311    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:15.175755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.175718    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"]
Oct 30 19:30:15.184712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.183978    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.200746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.199102    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d52364-openshift-default-dockercfg-4sxkj\""
Oct 30 19:30:15.204664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.204635    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"]
Oct 30 19:30:15.371712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.366878    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4md6t\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-kube-api-access-4md6t\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.371712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.366958    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-credential-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.371712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.367037    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-envoy\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.371712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.367084    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.371712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.367171    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7a4772-c323-4771-b082-59a9510df9f8-istio-podinfo\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.371712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.367229    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-certs\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.371712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.367256    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7a4772-c323-4771-b082-59a9510df9f8-istiod-ca-cert\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.371712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.367446    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-istio-token\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.371712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.367492    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-data\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.460764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.460327    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.470828    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-istio-token\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.470888    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-data\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.470994    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4md6t\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-kube-api-access-4md6t\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.471040    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-credential-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.471101    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-envoy\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.471139    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.471212    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7a4772-c323-4771-b082-59a9510df9f8-istio-podinfo\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.471260    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-certs\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.471285    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7a4772-c323-4771-b082-59a9510df9f8-istiod-ca-cert\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.472106    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7a4772-c323-4771-b082-59a9510df9f8-istiod-ca-cert\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.472681    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-data\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.478718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.473186    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-credential-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.482711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.480032    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.482711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.480454    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-certs\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.518709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.504272    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4md6t\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-kube-api-access-4md6t\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.520620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.520591    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-envoy\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.531422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.531395    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-istio-token\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.539291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.539263    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7a4772-c323-4771-b082-59a9510df9f8-istio-podinfo\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:15.674721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.664681    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:15.814019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:15.813932    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:16.091772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.088595    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:16.161789 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:30:16.158818    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7a4772_c323_4771_b082_59a9510df9f8.slice/crio-7d89727b2c10c8aad514a72ed3518e83ee3f9aee845bbf6f9e14b47ebf7c3e71 WatchSource:0}: Error finding container 7d89727b2c10c8aad514a72ed3518e83ee3f9aee845bbf6f9e14b47ebf7c3e71: Status 404 returned error can't find the container with id 7d89727b2c10c8aad514a72ed3518e83ee3f9aee845bbf6f9e14b47ebf7c3e71
Oct 30 19:30:16.165753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.162235    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"]
Oct 30 19:30:16.165753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.164155    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:16.165753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.164252    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:16.165753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.164299    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:16.255974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.255950    4925 scope.go:122] "RemoveContainer" containerID="7b2ee21b7cdf9ea9e765588ffa8ea39006ce31bb7e6e000c5b2aa06202d1052c"
Oct 30 19:30:16.256376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.256353    4925 scope.go:122] "RemoveContainer" containerID="24763723da445c603fea502b429f0ea4f4ef77ab2b413f2df511cf1734f2060e"
Oct 30 19:30:16.656105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.656070    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" event={"ID":"af7a4772-c323-4771-b082-59a9510df9f8","Type":"ContainerStarted","Data":"fe72b2c7b41d8936b0d601440861546412af3f353659c6955d0c312ebef8fa8f"}
Oct 30 19:30:16.656584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.656128    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" event={"ID":"af7a4772-c323-4771-b082-59a9510df9f8","Type":"ContainerStarted","Data":"7d89727b2c10c8aad514a72ed3518e83ee3f9aee845bbf6f9e14b47ebf7c3e71"}
Oct 30 19:30:16.680351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.680299    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" podStartSLOduration=1.6802860640000001 podStartE2EDuration="1.680286064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:16.677907752 +0000 UTC m=+1451.580920791" watchObservedRunningTime="2026-10-30 19:30:16.680286064 +0000 UTC m=+1451.583299101"
Oct 30 19:30:16.814183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.814149    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:16.822443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.822417    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"]
Oct 30 19:30:16.827147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.827124    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:16.832123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.832099    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-afdf16\"/\"test-model-afdf16-kserve-self-signed-certs\""
Oct 30 19:30:16.834510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.832581    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-afdf16\"/\"openshift-service-ca.crt\""
Oct 30 19:30:16.834510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.832679    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-afdf16\"/\"kube-root-ca.crt\""
Oct 30 19:30:16.834510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.832822    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-afdf16\"/\"default-dockercfg-hvvvx\""
Oct 30 19:30:16.862411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.843607    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"]
Oct 30 19:30:16.924246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.924217    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-tmp-dir\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:16.924435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.924387    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-model-cache\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:16.924516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.924444    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-dshm\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:16.924589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.924563    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-home\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:16.924772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.924753    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68f400a1-258e-4e0e-99ea-a440a7c12939-tls-certs\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:16.924848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.924830    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtpbc\" (UniqueName: \"kubernetes.io/projected/68f400a1-258e-4e0e-99ea-a440a7c12939-kube-api-access-jtpbc\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:16.977919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:16.977888    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:17.026805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.026775    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-model-cache\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.026967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.026822    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-dshm\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.026967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.026870    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-home\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.026967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.026953    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68f400a1-258e-4e0e-99ea-a440a7c12939-tls-certs\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.027102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.027009    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtpbc\" (UniqueName: \"kubernetes.io/projected/68f400a1-258e-4e0e-99ea-a440a7c12939-kube-api-access-jtpbc\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.027203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.027153    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-tmp-dir\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.027260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.027238    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-model-cache\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.027315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.027301    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-home\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.027649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.027632    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-tmp-dir\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.030005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.029980    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-dshm\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.030305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.030274    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68f400a1-258e-4e0e-99ea-a440a7c12939-tls-certs\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.039480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.039459    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtpbc\" (UniqueName: \"kubernetes.io/projected/68f400a1-258e-4e0e-99ea-a440a7c12939-kube-api-access-jtpbc\") pod \"test-model-afdf16-kserve-6db5d8b867-vjcbp\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") " pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.171730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.167231    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:17.178736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.176811    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:17.303727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.301216    4925 scope.go:122] "RemoveContainer" containerID="75c168dec6597f87fda3b68ce0c7852a62623ad0671762b498a147c7512089fe"
Oct 30 19:30:17.303727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.302065    4925 scope.go:122] "RemoveContainer" containerID="8d0f62ce0aff8570f771bfbeabb41b6061fd78bd040df3b8f0827e3dbcaebbf0"
Oct 30 19:30:17.651329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.651304    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"]
Oct 30 19:30:17.658681 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:30:17.658645    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68f400a1_258e_4e0e_99ea_a440a7c12939.slice/crio-4c58e8af833336e76806afe9689ef0113b29d103b7ef509160b3763c36c6716f WatchSource:0}: Error finding container 4c58e8af833336e76806afe9689ef0113b29d103b7ef509160b3763c36c6716f: Status 404 returned error can't find the container with id 4c58e8af833336e76806afe9689ef0113b29d103b7ef509160b3763c36c6716f
Oct 30 19:30:17.667343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.667319    4925 generic.go:356] "Generic (PLEG): container finished" podID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerID="fce3b126e5be5739467a4860aa18aa4c6e21da55e64035394775b96be9c1a972" exitCode=60
Oct 30 19:30:17.667447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.667392    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" event={"ID":"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949","Type":"ContainerDied","Data":"fce3b126e5be5739467a4860aa18aa4c6e21da55e64035394775b96be9c1a972"}
Oct 30 19:30:17.667531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.667461    4925 scope.go:122] "RemoveContainer" containerID="7b2ee21b7cdf9ea9e765588ffa8ea39006ce31bb7e6e000c5b2aa06202d1052c"
Oct 30 19:30:17.668134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.668112    4925 scope.go:122] "RemoveContainer" containerID="fce3b126e5be5739467a4860aa18aa4c6e21da55e64035394775b96be9c1a972"
Oct 30 19:30:17.668484 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:17.668467    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c_odh-ai-gateway-infra(1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949"
Oct 30 19:30:17.669963 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.669938    4925 generic.go:356] "Generic (PLEG): container finished" podID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerID="a0bd98ca2d68b194ce3edd10cbac4188c709f33a6195385ad1c09b80ab4d1a08" exitCode=60
Oct 30 19:30:17.670062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.669963    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" event={"ID":"6bd487ee-ef2f-4db0-a116-7b7650babfab","Type":"ContainerDied","Data":"a0bd98ca2d68b194ce3edd10cbac4188c709f33a6195385ad1c09b80ab4d1a08"}
Oct 30 19:30:17.670787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.670764    4925 scope.go:122] "RemoveContainer" containerID="a0bd98ca2d68b194ce3edd10cbac4188c709f33a6195385ad1c09b80ab4d1a08"
Oct 30 19:30:17.671129 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:17.671109    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-52rhp_odh-ai-gateway-infra(6bd487ee-ef2f-4db0-a116-7b7650babfab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab"
Oct 30 19:30:17.717279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.717251    4925 scope.go:122] "RemoveContainer" containerID="24763723da445c603fea502b429f0ea4f4ef77ab2b413f2df511cf1734f2060e"
Oct 30 19:30:17.786713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.786644    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:17.835539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:17.835511    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:18.682000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.681925    4925 generic.go:356] "Generic (PLEG): container finished" podID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerID="441b537c1d6c62dd93e397719777040a5545b13a7c4c86159d00e3a5fe9262ed" exitCode=60
Oct 30 19:30:18.682440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.682005    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" event={"ID":"5f28cc08-a010-45de-9449-1e8eb1d91dfc","Type":"ContainerDied","Data":"441b537c1d6c62dd93e397719777040a5545b13a7c4c86159d00e3a5fe9262ed"}
Oct 30 19:30:18.682440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.682065    4925 scope.go:122] "RemoveContainer" containerID="75c168dec6597f87fda3b68ce0c7852a62623ad0671762b498a147c7512089fe"
Oct 30 19:30:18.682552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.682473    4925 scope.go:122] "RemoveContainer" containerID="441b537c1d6c62dd93e397719777040a5545b13a7c4c86159d00e3a5fe9262ed"
Oct 30 19:30:18.682928 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:18.682899    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t_odh-ai-gateway-infra(5f28cc08-a010-45de-9449-1e8eb1d91dfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc"
Oct 30 19:30:18.687072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.687047    4925 generic.go:356] "Generic (PLEG): container finished" podID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerID="ef1c49dc30551d9456d7c8ed071097f326825cf9620127a0b784e06dd4301176" exitCode=60
Oct 30 19:30:18.687189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.687103    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" event={"ID":"5757f9f0-c294-4614-b5f9-c3cbb0e908e6","Type":"ContainerDied","Data":"ef1c49dc30551d9456d7c8ed071097f326825cf9620127a0b784e06dd4301176"}
Oct 30 19:30:18.687611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.687572    4925 scope.go:122] "RemoveContainer" containerID="ef1c49dc30551d9456d7c8ed071097f326825cf9620127a0b784e06dd4301176"
Oct 30 19:30:18.688050 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:18.688024    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w_odh-ai-gateway-infra(5757f9f0-c294-4614-b5f9-c3cbb0e908e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6"
Oct 30 19:30:18.728716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.728593    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp" event={"ID":"68f400a1-258e-4e0e-99ea-a440a7c12939","Type":"ContainerStarted","Data":"44422bd096a2a1d3c578fb3eb3357aec75724e43e36bef65e5b50dc5e0a1ee4c"}
Oct 30 19:30:18.728716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.728647    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp" event={"ID":"68f400a1-258e-4e0e-99ea-a440a7c12939","Type":"ContainerStarted","Data":"4c58e8af833336e76806afe9689ef0113b29d103b7ef509160b3763c36c6716f"}
Oct 30 19:30:18.728716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.728671    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:18.733010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.732919    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:30:18.733746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.733497    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:18.743910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.740865    4925 scope.go:122] "RemoveContainer" containerID="8d0f62ce0aff8570f771bfbeabb41b6061fd78bd040df3b8f0827e3dbcaebbf0"
Oct 30 19:30:18.743910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.743465    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"]
Oct 30 19:30:18.780135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.779940    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"]
Oct 30 19:30:18.854483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:18.854417    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp" podStartSLOduration=2.85439905 podStartE2EDuration="2.85439905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:18.784329331 +0000 UTC m=+1453.687342369" watchObservedRunningTime="2026-10-30 19:30:18.85439905 +0000 UTC m=+1453.757412088"
Oct 30 19:30:19.036813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.036751    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"]
Oct 30 19:30:19.043966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.043938    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c1c8043-kwf77"]
Oct 30 19:30:19.117487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.117457    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:19.183780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.183125    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:19.215030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.215007    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:19.281510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.281432    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03ac7df-c750-4d22-a756-0a0ee2e23d7d" path="/var/lib/kubelet/pods/b03ac7df-c750-4d22-a756-0a0ee2e23d7d/volumes"
Oct 30 19:30:19.310132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.310107    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-maas-api-service-ca\") pod \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\" (UID: \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\") "
Oct 30 19:30:19.310325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.310152    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-kube-api-access-8zgsg\" (UniqueName: \"kubernetes.io/projected/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-kube-api-access-8zgsg\") pod \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\" (UID: \"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949\") "
Oct 30 19:30:19.310507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.310476    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-maas-api-service-ca" pod "1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" (UID: "1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:19.310841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.310815    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:19.313024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.313002    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-kube-api-access-8zgsg" pod "1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" (UID: "1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949"). InnerVolumeSpecName "kube-api-access-8zgsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:19.313550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.313522    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:19.411613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.411580    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd487ee-ef2f-4db0-a116-7b7650babfab-kube-api-access-lnqm5\" (UniqueName: \"kubernetes.io/projected/6bd487ee-ef2f-4db0-a116-7b7650babfab-kube-api-access-lnqm5\") pod \"6bd487ee-ef2f-4db0-a116-7b7650babfab\" (UID: \"6bd487ee-ef2f-4db0-a116-7b7650babfab\") "
Oct 30 19:30:19.411803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.411789    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd487ee-ef2f-4db0-a116-7b7650babfab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd487ee-ef2f-4db0-a116-7b7650babfab-maas-api-service-ca\") pod \"6bd487ee-ef2f-4db0-a116-7b7650babfab\" (UID: \"6bd487ee-ef2f-4db0-a116-7b7650babfab\") "
Oct 30 19:30:19.412195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.412168    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd487ee-ef2f-4db0-a116-7b7650babfab-maas-api-service-ca" pod "6bd487ee-ef2f-4db0-a116-7b7650babfab" (UID: "6bd487ee-ef2f-4db0-a116-7b7650babfab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:19.412381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.412364    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zgsg\" (UniqueName: \"kubernetes.io/projected/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949-kube-api-access-8zgsg\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:19.412428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.412387    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd487ee-ef2f-4db0-a116-7b7650babfab-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:19.413783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.413757    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd487ee-ef2f-4db0-a116-7b7650babfab-kube-api-access-lnqm5" pod "6bd487ee-ef2f-4db0-a116-7b7650babfab" (UID: "6bd487ee-ef2f-4db0-a116-7b7650babfab"). InnerVolumeSpecName "kube-api-access-lnqm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:19.514622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.514590    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnqm5\" (UniqueName: \"kubernetes.io/projected/6bd487ee-ef2f-4db0-a116-7b7650babfab-kube-api-access-lnqm5\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:19.516096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.516065    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:19.604024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.603953    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:30:19.719972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.719941    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"]
Oct 30 19:30:19.732383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.732361    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"
Oct 30 19:30:19.732547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.732359    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c" event={"ID":"1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949","Type":"ContainerDied","Data":"27dcdb11fd5f312d093b5e303e7685aa38b42627f61c034b6b3e67bf0738b233"}
Oct 30 19:30:19.732547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.732496    4925 scope.go:122] "RemoveContainer" containerID="fce3b126e5be5739467a4860aa18aa4c6e21da55e64035394775b96be9c1a972"
Oct 30 19:30:19.735753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.735730    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"
Oct 30 19:30:19.735865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.735754    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp" event={"ID":"6bd487ee-ef2f-4db0-a116-7b7650babfab","Type":"ContainerDied","Data":"00efc1fc51305d4efedde57513f4215518d06b9160d0c60e04963b41249a1671"}
Oct 30 19:30:19.748972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.747741    4925 scope.go:122] "RemoveContainer" containerID="a0bd98ca2d68b194ce3edd10cbac4188c709f33a6195385ad1c09b80ab4d1a08"
Oct 30 19:30:19.764572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.764550    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"]
Oct 30 19:30:19.857800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.855263    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:19.863051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.863024    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"]
Oct 30 19:30:19.871785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.871588    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889810-zr58c"]
Oct 30 19:30:19.878723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.878398    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"]
Oct 30 19:30:19.895044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:19.885081    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-52rhp"]
Oct 30 19:30:20.042347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.042311    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:20.093337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.093302    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:20.138135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.138057    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-maas-api-service-ca\") pod \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\" (UID: \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\") "
Oct 30 19:30:20.138315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.138248    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f28cc08-a010-45de-9449-1e8eb1d91dfc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f28cc08-a010-45de-9449-1e8eb1d91dfc-maas-api-service-ca\") pod \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\" (UID: \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\") "
Oct 30 19:30:20.138413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.138391    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f28cc08-a010-45de-9449-1e8eb1d91dfc-kube-api-access-zd9jw\" (UniqueName: \"kubernetes.io/projected/5f28cc08-a010-45de-9449-1e8eb1d91dfc-kube-api-access-zd9jw\") pod \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\" (UID: \"5f28cc08-a010-45de-9449-1e8eb1d91dfc\") "
Oct 30 19:30:20.138482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.138451    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-maas-api-service-ca" pod "5757f9f0-c294-4614-b5f9-c3cbb0e908e6" (UID: "5757f9f0-c294-4614-b5f9-c3cbb0e908e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:20.138595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.138578    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-kube-api-access-qrcp5\" (UniqueName: \"kubernetes.io/projected/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-kube-api-access-qrcp5\") pod \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\" (UID: \"5757f9f0-c294-4614-b5f9-c3cbb0e908e6\") "
Oct 30 19:30:20.138807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.138782    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f28cc08-a010-45de-9449-1e8eb1d91dfc-maas-api-service-ca" pod "5f28cc08-a010-45de-9449-1e8eb1d91dfc" (UID: "5f28cc08-a010-45de-9449-1e8eb1d91dfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:20.139291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.139273    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.139423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.139409    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f28cc08-a010-45de-9449-1e8eb1d91dfc-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.141603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.141582    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f28cc08-a010-45de-9449-1e8eb1d91dfc-kube-api-access-zd9jw" pod "5f28cc08-a010-45de-9449-1e8eb1d91dfc" (UID: "5f28cc08-a010-45de-9449-1e8eb1d91dfc"). InnerVolumeSpecName "kube-api-access-zd9jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:20.141815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.141795    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-kube-api-access-qrcp5" pod "5757f9f0-c294-4614-b5f9-c3cbb0e908e6" (UID: "5757f9f0-c294-4614-b5f9-c3cbb0e908e6"). InnerVolumeSpecName "kube-api-access-qrcp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:20.241641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.241613    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd9jw\" (UniqueName: \"kubernetes.io/projected/5f28cc08-a010-45de-9449-1e8eb1d91dfc-kube-api-access-zd9jw\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.241641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.241640    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrcp5\" (UniqueName: \"kubernetes.io/projected/5757f9f0-c294-4614-b5f9-c3cbb0e908e6-kube-api-access-qrcp5\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:20.576620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.576580    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"]
Oct 30 19:30:20.577486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577461    4925 cpu_manager.go:414] "containerMap: removing container" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerName="cleanup"
Oct 30 19:30:20.577600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577485    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerName="cleanup"
Oct 30 19:30:20.577600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577511    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerName="cleanup"
Oct 30 19:30:20.577600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577520    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerName="cleanup"
Oct 30 19:30:20.577600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577529    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerName="cleanup"
Oct 30 19:30:20.577600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577536    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerName="cleanup"
Oct 30 19:30:20.577600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577558    4925 cpu_manager.go:414] "containerMap: removing container" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerName="cleanup"
Oct 30 19:30:20.577600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577565    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577618    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577625    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577638    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577644    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577655    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577661    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577677    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577699    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577709    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577716    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577732    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577739    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577909    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577922    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577932    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577950    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577964    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577975    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577986    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.577996    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerName="cleanup"
Oct 30 19:30:20.578058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.578004    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" containerName="cleanup"
Oct 30 19:30:20.582594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.582565    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:20.603625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.593280    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"]
Oct 30 19:30:20.603625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.594276    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d52364-dockercfg-29jbq\""
Oct 30 19:30:20.615996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.615939    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:20.648403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.648369    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq\" (UID: \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:20.648649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.648627    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776wg\" (UniqueName: \"kubernetes.io/projected/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-kube-api-access-776wg\") pod \"payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq\" (UID: \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:20.675098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.675035    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d52364\""
Oct 30 19:30:20.795066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.795021    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq\" (UID: \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:20.812229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.812197    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776wg\" (UniqueName: \"kubernetes.io/projected/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-kube-api-access-776wg\") pod \"payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq\" (UID: \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:20.824438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.810347    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:20.824806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.796406    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq\" (UID: \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:20.934720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.933771    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-776wg\" (UniqueName: \"kubernetes.io/projected/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-kube-api-access-776wg\") pod \"payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq\" (UID: \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:20.934720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.934361    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:20.961712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.958197    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w" event={"ID":"5757f9f0-c294-4614-b5f9-c3cbb0e908e6","Type":"ContainerDied","Data":"b50bbbf25c9618719da57274c8d4e440323862e44a777a17cf0fd9d81809f045"}
Oct 30 19:30:20.961712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.958268    4925 scope.go:122] "RemoveContainer" containerID="ef1c49dc30551d9456d7c8ed071097f326825cf9620127a0b784e06dd4301176"
Oct 30 19:30:20.961712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:20.958366    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"
Oct 30 19:30:21.047626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.047606    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"
Oct 30 19:30:21.054649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.047732    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t" event={"ID":"5f28cc08-a010-45de-9449-1e8eb1d91dfc","Type":"ContainerDied","Data":"466505fd159fc2ac3001e18571a942e150f1b7f5c4a0ebb5e28fdbc1cd3a01e7"}
Oct 30 19:30:21.058976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.058964    4925 scope.go:122] "RemoveContainer" containerID="441b537c1d6c62dd93e397719777040a5545b13a7c4c86159d00e3a5fe9262ed"
Oct 30 19:30:21.075462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.075415    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"]
Oct 30 19:30:21.082880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.082859    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889810-95b5w"]
Oct 30 19:30:21.109716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.108633    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"]
Oct 30 19:30:21.115928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.115894    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889810-6227t"]
Oct 30 19:30:21.266246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.266167    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" path="/var/lib/kubelet/pods/1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949/volumes"
Oct 30 19:30:21.266786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.266768    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" path="/var/lib/kubelet/pods/5757f9f0-c294-4614-b5f9-c3cbb0e908e6/volumes"
Oct 30 19:30:21.267245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.267231    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f28cc08-a010-45de-9449-1e8eb1d91dfc" path="/var/lib/kubelet/pods/5f28cc08-a010-45de-9449-1e8eb1d91dfc/volumes"
Oct 30 19:30:21.267978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.267964    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" path="/var/lib/kubelet/pods/6bd487ee-ef2f-4db0-a116-7b7650babfab/volumes"
Oct 30 19:30:21.368977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.368953    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"]
Oct 30 19:30:21.369940 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:30:21.369917    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d159211_5fdd_4e5f_8e50_034d2a4e3f4d.slice/crio-fae0da4c94b4403992bca5ced1a25e14fd8739e25d0a7b3acc282686589eabe0 WatchSource:0}: Error finding container fae0da4c94b4403992bca5ced1a25e14fd8739e25d0a7b3acc282686589eabe0: Status 404 returned error can't find the container with id fae0da4c94b4403992bca5ced1a25e14fd8739e25d0a7b3acc282686589eabe0
Oct 30 19:30:21.440210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:21.440176    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:22.050096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:22.050057    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:22.059165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:22.059134    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq" event={"ID":"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d","Type":"ContainerStarted","Data":"7b204e6d5f944e5b23cf5b1019624330ba38ddde820aa36926ef9fbfb6ed2b5f"}
Oct 30 19:30:22.059414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:22.059385    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq" event={"ID":"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d","Type":"ContainerStarted","Data":"fae0da4c94b4403992bca5ced1a25e14fd8739e25d0a7b3acc282686589eabe0"}
Oct 30 19:30:22.059599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:22.059582    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:22.078550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:22.078499    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq" podStartSLOduration=2.078484922 podStartE2EDuration="2.078484922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:22.076958806 +0000 UTC m=+1456.979971843" watchObservedRunningTime="2026-10-30 19:30:22.078484922 +0000 UTC m=+1456.981497962"
Oct 30 19:30:22.126984 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:22.126947    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:30:22.380606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:22.380512    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:22.548524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:22.548490    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:22.722865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:22.722831    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:23.869341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:23.869307    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:24.601365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:24.601335    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:30:24.671323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:24.671289    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:25.766664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:25.766630    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:26.932869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:26.932811    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:27.193389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:27.192245    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:27.404419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:27.404392    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:28.126401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:28.126360    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:28.317986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:28.317906    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:28.509769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:28.509737    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:28.562344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:28.562299    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:30:28.808017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:28.807912    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:28.928978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:28.928932    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.000943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:28.992869    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.202715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.198001    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.369015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.368966    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.428753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.428205    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:29.482381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.482275    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.569859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.569792    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp" podUID="68f400a1-258e-4e0e-99ea-a440a7c12939" containerName="main" probeResult="failure" output="Get \"https://10.132.0.150:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.599824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.599776    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.699712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.676584    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:29.800818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.784020    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.812545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.812508    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:30:29.817713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.813599    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:30:29.874425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:29.874383    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.041713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.015844    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.072728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.072621    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:30:30.219216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.217113    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:30:30.401732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.400571    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:30:30.406900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.406863    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:30:30.592713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.592493    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:30:30.640409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.640362    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.658112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.647605    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:30:30.683177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.683150    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:30:30.725027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.725000    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:30:30.744676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.744649    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:30:30.973603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:30.973491    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:32.037836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:32.037801    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:33.080279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:33.080246    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:30:33.120725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:33.120359    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:33.896711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:33.894319    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:34.949941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:34.947397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:35.554086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:35.553928    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:35.590829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:35.590797    4925 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:30:35.591009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:35.590852    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:36.506914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:36.506844    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:37.258161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:37.258125    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:37.871809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:37.871772    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:38.464930 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:38.464897    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb24c7dd_850b_40a4_b25b_2450e5fffe39.slice/crio-conmon-eacfe6ef81a7811d3b8141d985b42a866e9dcccf04200f89c89022dd13d2dce1.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:30:38.531270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:38.531191    4925 generic.go:356] "Generic (PLEG): container finished" podID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerID="eacfe6ef81a7811d3b8141d985b42a866e9dcccf04200f89c89022dd13d2dce1" exitCode=0
Oct 30 19:30:38.531427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:38.531275    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" event={"ID":"fb24c7dd-850b-40a4-b25b-2450e5fffe39","Type":"ContainerDied","Data":"eacfe6ef81a7811d3b8141d985b42a866e9dcccf04200f89c89022dd13d2dce1"}
Oct 30 19:30:38.531791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:38.531737    4925 scope.go:122] "RemoveContainer" containerID="eacfe6ef81a7811d3b8141d985b42a866e9dcccf04200f89c89022dd13d2dce1"
Oct 30 19:30:38.533447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:38.533414    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:38.533584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:38.533505    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:38.533584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:38.533562    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:38.626204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:38.626171    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:39.085591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:39.085506    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:39.366670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:39.366590    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:39.540783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:39.540744    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" event={"ID":"fb24c7dd-850b-40a4-b25b-2450e5fffe39","Type":"ContainerStarted","Data":"fb9b40aa9f8acd05216f2bad926a481b15dcffe97505042354aa13772daa8efa"}
Oct 30 19:30:39.877156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:39.877124    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:30:39.877372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:39.877303    4925 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:30:39.877372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:39.877358    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:30:40.282745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:40.279577    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:40.877930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:40.877898    4925 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:30:40.878129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:40.877962    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:30:41.237291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:41.237240    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:41.893796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:41.893758    4925 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:30:41.894434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:41.894400    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:30:42.750787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:42.750757    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:42.878045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:42.878008    4925 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:30:42.878210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:42.878073    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:30:43.682409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:43.682374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:43.857461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:43.857432    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:43.877996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:43.877965    4925 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:30:43.878167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:43.878021    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:30:44.322833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:44.322799    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:44.878062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:44.878027    4925 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:30:44.878454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:44.878100    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:30:44.956353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:44.956058    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:45.838825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:45.838779    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:45.882363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:45.882334    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:30:45.882968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:45.882948    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:30:45.883748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:45.883725    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:30:46.474314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:46.474274    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:47.396343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:47.396288    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:47.991625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:47.991592    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:48.764298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:48.760079    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.306077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:49.306044    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.487571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:49.487536    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.727415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:49.727381    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"]
Oct 30 19:30:49.727907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:49.727874    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy" containerID="cri-o://857dd413838715c45ae16ebe278b430634597542facb04734b9ae911a3ec3a0b" gracePeriod=30
Oct 30 19:30:49.878749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:49.878714    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:50.813478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:50.813452    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:51.349738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:51.349706    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:51.586737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:51.586700    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:51.764316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:51.764281    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:51.929905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:51.929872    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:52.130709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:52.129856    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:52.142185 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:30:52.142140    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:30:52.986759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:52.986723    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:53.689275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:53.689243    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:54.471498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:54.471467    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:54.898492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:54.898458    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:55.216504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.216470    4925 generic.go:356] "Generic (PLEG): container finished" podID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerID="857dd413838715c45ae16ebe278b430634597542facb04734b9ae911a3ec3a0b" exitCode=0
Oct 30 19:30:55.217049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.217024    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" event={"ID":"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9","Type":"ContainerDied","Data":"857dd413838715c45ae16ebe278b430634597542facb04734b9ae911a3ec3a0b"}
Oct 30 19:30:55.217277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.217263    4925 scope.go:122] "RemoveContainer" containerID="04ad9f219c6fbe81c0c26e0679841c4f2144de70c979e0ccadc6fc559a6d6234"
Oct 30 19:30:55.345758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.345734    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:30:55.401728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.401257    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:55.434894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.434860    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-token\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-token\") pod \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") "
Oct 30 19:30:55.435106 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.434924    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-certs\") pod \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") "
Oct 30 19:30:55.435183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435126    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istiod-ca-cert\") pod \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") "
Oct 30 19:30:55.435183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435170    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-podinfo\") pod \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") "
Oct 30 19:30:55.435334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435237    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-envoy\") pod \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") "
Oct 30 19:30:55.435334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435272    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-credential-socket\") pod \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") "
Oct 30 19:30:55.435334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435286    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-certs" pod "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" (UID: "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:55.435493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435352    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-data\") pod \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") "
Oct 30 19:30:55.435493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435401    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-kube-api-access-9hbc8\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-kube-api-access-9hbc8\") pod \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") "
Oct 30 19:30:55.435600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435565    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-credential-socket" pod "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" (UID: "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:55.435653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435594    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istiod-ca-cert" pod "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" (UID: "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:55.435746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.435665    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-socket\") pod \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\" (UID: \"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9\") "
Oct 30 19:30:55.436108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.436007    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-data" pod "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" (UID: "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:55.436330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.436308    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-socket" pod "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" (UID: "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:55.436668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.436647    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:55.436792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.436674    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:55.436792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.436707    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:55.436792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.436720    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:55.436792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.436732    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:55.438763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.438674    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-podinfo" pod "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" (UID: "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:30:55.439014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.438988    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-envoy" pod "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" (UID: "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:55.443198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.443172    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-kube-api-access-9hbc8" pod "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" (UID: "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9"). InnerVolumeSpecName "kube-api-access-9hbc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:55.445759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.445732    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-token" pod "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" (UID: "a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:55.539045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.538972    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hbc8\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-kube-api-access-9hbc8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:55.539045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.539002    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:55.539045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.539012    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:55.539045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:55.539022    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.023195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.023134    4925 patch_prober.go:29] interesting pod/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:30:56.023388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.023240    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:56.158195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.158148    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:56.228559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.228531    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"
Oct 30 19:30:56.228768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.228567    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr" event={"ID":"a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9","Type":"ContainerDied","Data":"4ed97341a30657bd7d2d1ca73c9e4ddd393001430a5914fd45f7c73194cdf445"}
Oct 30 19:30:56.228768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.228625    4925 scope.go:122] "RemoveContainer" containerID="857dd413838715c45ae16ebe278b430634597542facb04734b9ae911a3ec3a0b"
Oct 30 19:30:56.256874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.256844    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"]
Oct 30 19:30:56.261380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.261359    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8ceda0-openshift-default-6849d5bcf-gcffr"]
Oct 30 19:30:56.344565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.344490    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:56.630114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:56.630032    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:57.125130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:57.125087    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:57.319712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:57.309222    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:57.502947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:57.497291    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" path="/var/lib/kubelet/pods/a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9/volumes"
Oct 30 19:30:58.072730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:58.072668    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:58.764119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:58.764080    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:59.522992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:30:59.522958    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:00.208665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:00.208632    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:00.459920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:00.459845    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"]
Oct 30 19:31:00.470599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:00.470568    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"]
Oct 30 19:31:00.587750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:00.587679    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:00.762796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:00.758242    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:01.158776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:01.158744    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:01.530389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:01.530284    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:01.694534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:01.694499    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:02.295431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:02.295394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:02.481015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:02.480980    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:02.646724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:02.645345    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:03.524712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:03.514539    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:04.592575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:04.592539    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:05.387822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:05.387737    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:05.463066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:05.463034    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f" podUID="9a049015-a805-45ec-bbe8-786155bab0bb" containerName="payload-processing" containerID="cri-o://3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662" gracePeriod=25
Oct 30 19:31:05.472938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:05.472878    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx" podUID="ba76082c-f99e-47f2-8d57-0283bfb7e872" containerName="payload-pre-processing" containerID="cri-o://46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff" gracePeriod=25
Oct 30 19:31:05.913521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:05.913431    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:06.093712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:06.080818    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:31:06.122536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:06.122508    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:31:06.543153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:06.543064    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:06.861777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:06.848641    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:06.941079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:06.941024    4925 generic.go:356] "Generic (PLEG): container finished" podID="9a049015-a805-45ec-bbe8-786155bab0bb" containerID="3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662" exitCode=0
Oct 30 19:31:06.962945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:06.962902    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f" event={"ID":"9a049015-a805-45ec-bbe8-786155bab0bb","Type":"ContainerDied","Data":"3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662"}
Oct 30 19:31:07.021219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.021187    4925 generic.go:356] "Generic (PLEG): container finished" podID="ba76082c-f99e-47f2-8d57-0283bfb7e872" containerID="46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff" exitCode=0
Oct 30 19:31:07.025828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.025795    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx" event={"ID":"ba76082c-f99e-47f2-8d57-0283bfb7e872","Type":"ContainerDied","Data":"46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff"}
Oct 30 19:31:07.193137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.193108    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:07.215706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.213060    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:31:07.373154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.373124    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba76082c-f99e-47f2-8d57-0283bfb7e872-kube-api-access-hn5zd\" (UniqueName: \"kubernetes.io/projected/ba76082c-f99e-47f2-8d57-0283bfb7e872-kube-api-access-hn5zd\") pod \"ba76082c-f99e-47f2-8d57-0283bfb7e872\" (UID: \"ba76082c-f99e-47f2-8d57-0283bfb7e872\") "
Oct 30 19:31:07.373541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.373521    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba76082c-f99e-47f2-8d57-0283bfb7e872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba76082c-f99e-47f2-8d57-0283bfb7e872-plugins-config-volume\") pod \"ba76082c-f99e-47f2-8d57-0283bfb7e872\" (UID: \"ba76082c-f99e-47f2-8d57-0283bfb7e872\") "
Oct 30 19:31:07.375148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.375122    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba76082c-f99e-47f2-8d57-0283bfb7e872-plugins-config-volume" pod "ba76082c-f99e-47f2-8d57-0283bfb7e872" (UID: "ba76082c-f99e-47f2-8d57-0283bfb7e872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:07.391751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.389534    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:31:07.524290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.524244    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:07.566791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.543069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:07.575625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.572556    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba76082c-f99e-47f2-8d57-0283bfb7e872-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:07.604718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.586518    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:07.733444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.733385    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a049015-a805-45ec-bbe8-786155bab0bb-kube-api-access-k4bsj\" (UniqueName: \"kubernetes.io/projected/9a049015-a805-45ec-bbe8-786155bab0bb-kube-api-access-k4bsj\") pod \"9a049015-a805-45ec-bbe8-786155bab0bb\" (UID: \"9a049015-a805-45ec-bbe8-786155bab0bb\") "
Oct 30 19:31:07.733444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.733447    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a049015-a805-45ec-bbe8-786155bab0bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a049015-a805-45ec-bbe8-786155bab0bb-plugins-config-volume\") pod \"9a049015-a805-45ec-bbe8-786155bab0bb\" (UID: \"9a049015-a805-45ec-bbe8-786155bab0bb\") "
Oct 30 19:31:07.768716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.754950    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a049015-a805-45ec-bbe8-786155bab0bb-plugins-config-volume" pod "9a049015-a805-45ec-bbe8-786155bab0bb" (UID: "9a049015-a805-45ec-bbe8-786155bab0bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:07.768716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.757055    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:31:07.768716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.758144    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:07.768716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.760304    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:31:07.826955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.826921    4925 scope.go:122] "RemoveContainer" containerID="e25805366ec6e869a26ebb2484487be0f32c99e76734ce8dee84f195424888f3"
Oct 30 19:31:07.838369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.838334    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a049015-a805-45ec-bbe8-786155bab0bb-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:07.887930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.887890    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba76082c-f99e-47f2-8d57-0283bfb7e872-kube-api-access-hn5zd" pod "ba76082c-f99e-47f2-8d57-0283bfb7e872" (UID: "ba76082c-f99e-47f2-8d57-0283bfb7e872"). InnerVolumeSpecName "kube-api-access-hn5zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:07.927722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.925727    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a049015-a805-45ec-bbe8-786155bab0bb-kube-api-access-k4bsj" pod "9a049015-a805-45ec-bbe8-786155bab0bb" (UID: "9a049015-a805-45ec-bbe8-786155bab0bb"). InnerVolumeSpecName "kube-api-access-k4bsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:07.943846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.943278    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5zd\" (UniqueName: \"kubernetes.io/projected/ba76082c-f99e-47f2-8d57-0283bfb7e872-kube-api-access-hn5zd\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:07.943846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.943313    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4bsj\" (UniqueName: \"kubernetes.io/projected/9a049015-a805-45ec-bbe8-786155bab0bb-kube-api-access-k4bsj\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:07.952608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:07.952558    4925 scope.go:122] "RemoveContainer" containerID="163e52568b64445dd1466cd5ec7a1b5731f4bcfbd6e7175e282e042a20a397d6"
Oct 30 19:31:08.012060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.007865    4925 scope.go:122] "RemoveContainer" containerID="507e5a21945e641e3fd73d81234c016a5bbaa707412b5c24689b38c1cfe34ad5"
Oct 30 19:31:08.090719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.090573    4925 scope.go:122] "RemoveContainer" containerID="579025e32da6698e02dcbc4a60a155a24cc45173b391dcbb42336a868b361e5e"
Oct 30 19:31:08.102715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.099148    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx" event={"ID":"ba76082c-f99e-47f2-8d57-0283bfb7e872","Type":"ContainerDied","Data":"4dfccffb7ea9e94d2700d323153491bbc01f1c90b6d85d4a6434e2fa0b576320"}
Oct 30 19:31:08.102715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.099212    4925 scope.go:122] "RemoveContainer" containerID="46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff"
Oct 30 19:31:08.102715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.099371    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"
Oct 30 19:31:08.134556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.129830    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f" event={"ID":"9a049015-a805-45ec-bbe8-786155bab0bb","Type":"ContainerDied","Data":"e25d87fbe1a2483134c3b9f9f73fc7de9cf466ae2c8a55ddf048a1bf3bb677a8"}
Oct 30 19:31:08.134556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.129981    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"
Oct 30 19:31:08.167241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.162269    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"]
Oct 30 19:31:08.190948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.188630    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f61e62-bdcf9d7f9-mjqtx"]
Oct 30 19:31:08.257156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.228235    4925 scope.go:122] "RemoveContainer" containerID="3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662"
Oct 30 19:31:08.283702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.283651    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"]
Oct 30 19:31:08.290830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.290351    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f61e62-5d44b7544d-zkr4f"]
Oct 30 19:31:08.291088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.291066    4925 scope.go:122] "RemoveContainer" containerID="d4a03abdaef1973f9bf19141fab73482f78caa334a76ac27999fce5d5e95e6af"
Oct 30 19:31:08.341892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.341858    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:08.377046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:08.376341    4925 scope.go:122] "RemoveContainer" containerID="3c76df8830ccfcbf5fe3fae63c7fe4eeb5db7db7a876f4a7bd4c39fe09e896c2"
Oct 30 19:31:09.007711 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:09.002555    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:09.195850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:09.195811    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:09.345026 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:09.340289    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:09.392044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:09.391987    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a049015-a805-45ec-bbe8-786155bab0bb" path="/var/lib/kubelet/pods/9a049015-a805-45ec-bbe8-786155bab0bb/volumes"
Oct 30 19:31:09.392868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:09.392844    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba76082c-f99e-47f2-8d57-0283bfb7e872" path="/var/lib/kubelet/pods/ba76082c-f99e-47f2-8d57-0283bfb7e872/volumes"
Oct 30 19:31:10.119714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:10.119601    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:11.027712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:11.027661    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:11.442714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:11.436013    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:11.605751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:11.605720    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:12.148962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:12.148931    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:12.686877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:12.686845    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:12.867181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:12.867148    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:13.058494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.058385    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:13.610615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.610584    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"]
Oct 30 19:31:13.611378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611358    4925 cpu_manager.go:414] "containerMap: removing container" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy"
Oct 30 19:31:13.611485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611380    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy"
Oct 30 19:31:13.611485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611400    4925 cpu_manager.go:414] "containerMap: removing container" podUID="ba76082c-f99e-47f2-8d57-0283bfb7e872" containerName="payload-pre-processing"
Oct 30 19:31:13.611485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611406    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba76082c-f99e-47f2-8d57-0283bfb7e872" containerName="payload-pre-processing"
Oct 30 19:31:13.611485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611437    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9a049015-a805-45ec-bbe8-786155bab0bb" containerName="payload-processing"
Oct 30 19:31:13.611485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611444    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a049015-a805-45ec-bbe8-786155bab0bb" containerName="payload-processing"
Oct 30 19:31:13.611485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611455    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerName="cleanup"
Oct 30 19:31:13.611485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611461    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerName="cleanup"
Oct 30 19:31:13.611485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611480    4925 cpu_manager.go:414] "containerMap: removing container" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerName="cleanup"
Oct 30 19:31:13.611485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611488    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerName="cleanup"
Oct 30 19:31:13.612029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611505    4925 cpu_manager.go:414] "containerMap: removing container" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy"
Oct 30 19:31:13.612029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611514    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy"
Oct 30 19:31:13.612029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611714    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a049015-a805-45ec-bbe8-786155bab0bb" containerName="payload-processing"
Oct 30 19:31:13.612029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611728    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd487ee-ef2f-4db0-a116-7b7650babfab" containerName="cleanup"
Oct 30 19:31:13.612029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611739    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5757f9f0-c294-4614-b5f9-c3cbb0e908e6" containerName="cleanup"
Oct 30 19:31:13.612029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611752    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy"
Oct 30 19:31:13.612029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611767    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebaa96b-e5f3-4479-aa0d-ba7ffb1ae949" containerName="cleanup"
Oct 30 19:31:13.612029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.611784    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba76082c-f99e-47f2-8d57-0283bfb7e872" containerName="payload-pre-processing"
Oct 30 19:31:13.618620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.618594    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.631493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.629310    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef5dca\"/\"test-model-ef5dca-kserve-self-signed-certs\""
Oct 30 19:31:13.631493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.629386    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef5dca\"/\"kube-root-ca.crt\""
Oct 30 19:31:13.631493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.629456    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef5dca\"/\"openshift-service-ca.crt\""
Oct 30 19:31:13.631493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.629567    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef5dca\"/\"default-dockercfg-jm7ln\""
Oct 30 19:31:13.631875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.631601    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"]
Oct 30 19:31:13.695318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.694544    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-dshm\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.695318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.694634    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tlm8\" (UniqueName: \"kubernetes.io/projected/3f1076a2-62c1-4765-b1f0-6465fc85c599-kube-api-access-9tlm8\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.695318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.694728    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-model-cache\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.695318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.694806    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-home\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.695318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.694863    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1076a2-62c1-4765-b1f0-6465fc85c599-tls-certs\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.695318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.695008    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-tmp-dir\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.700622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.700590    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:13.806144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.796946    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tlm8\" (UniqueName: \"kubernetes.io/projected/3f1076a2-62c1-4765-b1f0-6465fc85c599-kube-api-access-9tlm8\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.806144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.797025    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-model-cache\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.806144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.797107    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-home\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.806144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.797168    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1076a2-62c1-4765-b1f0-6465fc85c599-tls-certs\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.806144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.797326    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-tmp-dir\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.806144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.797519    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-dshm\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.806144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.803175    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-home\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.806999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.806972    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-tmp-dir\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.821809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.821771    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-model-cache\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.861737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.828296    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1076a2-62c1-4765-b1f0-6465fc85c599-tls-certs\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.861737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.838498    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tlm8\" (UniqueName: \"kubernetes.io/projected/3f1076a2-62c1-4765-b1f0-6465fc85c599-kube-api-access-9tlm8\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.894718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.879165    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-dshm\") pod \"test-model-ef5dca-kserve-7696c4f855-98htn\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") " pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.944537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.944192    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:13.975734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:13.970305    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:14.435721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:14.434899    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"]
Oct 30 19:31:14.446292 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:31:14.446217    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f1076a2_62c1_4765_b1f0_6465fc85c599.slice/crio-a04dc18b84c50eab753ede345083f9b2f38ea86db383a21002f1a99db96c6fb8 WatchSource:0}: Error finding container a04dc18b84c50eab753ede345083f9b2f38ea86db383a21002f1a99db96c6fb8: Status 404 returned error can't find the container with id a04dc18b84c50eab753ede345083f9b2f38ea86db383a21002f1a99db96c6fb8
Oct 30 19:31:14.858877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:14.858783    4925 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:31:14.861771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:14.860031    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:14.866637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:14.864083    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:14.931954 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:14.931916    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:14.931954 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:14.931937    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:15.399106 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:15.398509    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn" event={"ID":"3f1076a2-62c1-4765-b1f0-6465fc85c599","Type":"ContainerStarted","Data":"01bc6a21766a80dff75f8f01c2cbe1260474d6220013bad861899eea5310eb66"}
Oct 30 19:31:15.399106 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:15.398558    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn" event={"ID":"3f1076a2-62c1-4765-b1f0-6465fc85c599","Type":"ContainerStarted","Data":"a04dc18b84c50eab753ede345083f9b2f38ea86db383a21002f1a99db96c6fb8"}
Oct 30 19:31:15.424089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:15.424025    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn" podStartSLOduration=2.4240056 podStartE2EDuration="2.4240056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:15.418081173 +0000 UTC m=+1510.321094211" watchObservedRunningTime="2026-10-30 19:31:15.4240056 +0000 UTC m=+1510.327018620"
Oct 30 19:31:15.557552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:15.557520    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:15.945046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:15.945017    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:16.367613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:16.362079    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:17.036554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:17.028674    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:17.785047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:17.784351    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:18.620538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:18.620502    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:19.227764 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:19.085849    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:19.253597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.253027    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": read tcp 10.132.0.2:52662->10.132.0.149:15021: read: connection reset by peer" start-of-body=
Oct 30 19:31:19.254720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.253651    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": read tcp 10.132.0.2:52662->10.132.0.149:15021: read: connection reset by peer"
Oct 30 19:31:19.338203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.338168    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:19.440095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.440065    4925 generic.go:356] "Generic (PLEG): container finished" podID="af7a4772-c323-4771-b082-59a9510df9f8" containerID="fe72b2c7b41d8936b0d601440861546412af3f353659c6955d0c312ebef8fa8f" exitCode=0
Oct 30 19:31:19.440260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.440148    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" event={"ID":"af7a4772-c323-4771-b082-59a9510df9f8","Type":"ContainerDied","Data":"fe72b2c7b41d8936b0d601440861546412af3f353659c6955d0c312ebef8fa8f"}
Oct 30 19:31:19.440725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.440700    4925 scope.go:122] "RemoveContainer" containerID="fe72b2c7b41d8936b0d601440861546412af3f353659c6955d0c312ebef8fa8f"
Oct 30 19:31:19.442073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.442053    4925 generic.go:356] "Generic (PLEG): container finished" podID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerID="1cd424c0dc77ab3dda6417736cb8e6cc04aa157d640a119f13f08e2116839064" exitCode=0
Oct 30 19:31:19.442146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.442126    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" event={"ID":"b5a0d956-67cf-40d2-a0b0-427786ce6161","Type":"ContainerDied","Data":"1cd424c0dc77ab3dda6417736cb8e6cc04aa157d640a119f13f08e2116839064"}
Oct 30 19:31:19.442557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.442543    4925 scope.go:122] "RemoveContainer" containerID="1cd424c0dc77ab3dda6417736cb8e6cc04aa157d640a119f13f08e2116839064"
Oct 30 19:31:19.442969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.442940    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:31:19.443080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.443010    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:31:19.443080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.443043    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:31:19.444555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.444530    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:31:19.444631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.444599    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:31:19.444668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:19.444631    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:31:20.073712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.072265    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:20.248870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.248840    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:20.469725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.469315    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" event={"ID":"af7a4772-c323-4771-b082-59a9510df9f8","Type":"ContainerStarted","Data":"0c2d65904d793c8509695edead5f402e63a34f4ed61f7109d43cd1cefba2a4c5"}
Oct 30 19:31:20.473871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.473843    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" event={"ID":"b5a0d956-67cf-40d2-a0b0-427786ce6161","Type":"ContainerStarted","Data":"18bc57ad37fe73844e365a6529054189c023f9b0f24d6c263c0ca85efe1c3443"}
Oct 30 19:31:20.492900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.492787    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:20.600712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.593930    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:31:20.606023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.605991    4925 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:31:20.606265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.606240    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:31:20.816767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.816119    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:31:20.816767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.816270    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:20.816767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:20.816313    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:21.238735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:21.236217    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:21.547186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:21.547092    4925 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:31:21.547466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:21.547439    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:31:21.842772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:21.839280    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:21.842772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:21.839347    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:21.864404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:21.864371    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:22.172714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:22.172189    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:22.354711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:22.354504    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:22.475196 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:22.470617    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:31:22.548313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:22.548279    4925 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:31:22.548576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:22.548549    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:31:22.754255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:22.753637    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:22.817714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:22.815134    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:22.817714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:22.815192    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:23.437717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:23.430988    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:23.558713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:23.553203    4925 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:31:23.558713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:23.553268    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:31:23.815772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:23.815085    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:23.815772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:23.815167    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:23.947675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:23.947106    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:23.964138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:23.964108    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:24.381587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:24.381551    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:24.545947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:24.545488    4925 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:31:24.545947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:24.545546    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:31:24.561797 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:24.561755    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:24.593322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:24.593291    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:31:24.831718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:24.820042    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:24.831718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:24.820099    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:25.059713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:25.057365    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:25.546714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:25.546188    4925 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:31:25.546714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:25.546257    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:31:25.816277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:25.810064    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:25.816277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:25.815331    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:25.816277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:25.815515    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:26.465714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:26.463072    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:26.546268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:26.546233    4925 patch_prober.go:29] interesting pod/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:31:26.546435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:26.546310    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:31:26.823494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:26.821852    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:31:26.823494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:26.822767    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:31:26.824290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:26.824269    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:31:27.158714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:27.154038    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:27.320540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:27.320515    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:27.525034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:27.524931    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:27.551566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:27.551540    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:31:27.552030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:27.552015    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:31:27.552838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:27.552817    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:31:27.882153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:27.882054    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:28.556733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:28.555957    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:29.101733 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:29.101680    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:29.566160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:29.566125    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:30.481043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:30.480998    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:30.659033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:30.659004    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:31.100307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:31.100269    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:31.885204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:31.885126    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:32.076194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:32.076170    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:32.443712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:32.443351    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:33.195651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:33.195612    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:33.550518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:33.550421    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:33.762712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:33.755489    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:34.116710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:34.115788    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:34.699411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:34.699376    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:35.546468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:35.546433    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:36.162279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:36.162234    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:36.175416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:36.175390    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"]
Oct 30 19:31:36.175608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:36.175586    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy" containerID="cri-o://9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856" gracePeriod=30
Oct 30 19:31:36.483325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:36.483210    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:37.230972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:37.228213    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:37.403172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:37.403095    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:37.781491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:37.781456    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:38.310160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:38.310124    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:38.594723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:38.594619    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:39.159474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:39.159440    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:39.197095 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:39.197028    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:39.292278 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:39.292197    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:39.552024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:39.551943    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:39.735308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:39.735270    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:40.378786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:40.378745    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:41.289323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.289207    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:41.492166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.492140    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:31:41.596393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596362    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-credential-socket\") pod \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") "
Oct 30 19:31:41.596571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596417    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-certs\") pod \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") "
Oct 30 19:31:41.596571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596508    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-kube-api-access-6np5b\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-kube-api-access-6np5b\") pod \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") "
Oct 30 19:31:41.596661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596590    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-envoy\") pod \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") "
Oct 30 19:31:41.596745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596722    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-certs" pod "8999c9ff-9ed0-4705-93e5-6bca3a32ed68" (UID: "8999c9ff-9ed0-4705-93e5-6bca3a32ed68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:41.596811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596741    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-socket\") pod \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") "
Oct 30 19:31:41.596811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596795    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-data\") pod \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") "
Oct 30 19:31:41.596924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596845    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-podinfo\") pod \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") "
Oct 30 19:31:41.596924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596875    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-credential-socket" pod "8999c9ff-9ed0-4705-93e5-6bca3a32ed68" (UID: "8999c9ff-9ed0-4705-93e5-6bca3a32ed68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:41.596924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596890    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istiod-ca-cert\") pod \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") "
Oct 30 19:31:41.597092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.596924    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-token\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-token\") pod \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\" (UID: \"8999c9ff-9ed0-4705-93e5-6bca3a32ed68\") "
Oct 30 19:31:41.597144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.597109    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-socket" pod "8999c9ff-9ed0-4705-93e5-6bca3a32ed68" (UID: "8999c9ff-9ed0-4705-93e5-6bca3a32ed68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:41.597437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.597414    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istiod-ca-cert" pod "8999c9ff-9ed0-4705-93e5-6bca3a32ed68" (UID: "8999c9ff-9ed0-4705-93e5-6bca3a32ed68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:41.597723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.597699    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.597837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.597727    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.597837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.597741    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.597837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.597756    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.598015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.597916    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-data" pod "8999c9ff-9ed0-4705-93e5-6bca3a32ed68" (UID: "8999c9ff-9ed0-4705-93e5-6bca3a32ed68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:41.599402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.599367    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-envoy" pod "8999c9ff-9ed0-4705-93e5-6bca3a32ed68" (UID: "8999c9ff-9ed0-4705-93e5-6bca3a32ed68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:41.599930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.599903    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-token" pod "8999c9ff-9ed0-4705-93e5-6bca3a32ed68" (UID: "8999c9ff-9ed0-4705-93e5-6bca3a32ed68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:41.600174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.600146    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-podinfo" pod "8999c9ff-9ed0-4705-93e5-6bca3a32ed68" (UID: "8999c9ff-9ed0-4705-93e5-6bca3a32ed68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:31:41.600394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.600374    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-kube-api-access-6np5b" pod "8999c9ff-9ed0-4705-93e5-6bca3a32ed68" (UID: "8999c9ff-9ed0-4705-93e5-6bca3a32ed68"). InnerVolumeSpecName "kube-api-access-6np5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:41.699674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.699640    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.699674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.699672    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.699903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.699698    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.699903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.699714    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.699903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.699727    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6np5b\" (UniqueName: \"kubernetes.io/projected/8999c9ff-9ed0-4705-93e5-6bca3a32ed68-kube-api-access-6np5b\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.964408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:41.964308    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:42.001637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.001605    4925 generic.go:356] "Generic (PLEG): container finished" podID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerID="9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856" exitCode=0
Oct 30 19:31:42.001901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.001655    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"
Oct 30 19:31:42.001901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.001707    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" event={"ID":"8999c9ff-9ed0-4705-93e5-6bca3a32ed68","Type":"ContainerDied","Data":"9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856"}
Oct 30 19:31:42.001901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.001776    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj" event={"ID":"8999c9ff-9ed0-4705-93e5-6bca3a32ed68","Type":"ContainerDied","Data":"cdf08ff0880ff550e4c8655f1a9a1707d58c938bdf076bc4136f49e5b8262e14"}
Oct 30 19:31:42.001901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.001807    4925 scope.go:122] "RemoveContainer" containerID="9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856"
Oct 30 19:31:42.028478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.028449    4925 scope.go:122] "RemoveContainer" containerID="9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856"
Oct 30 19:31:42.029022 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:42.028957    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856\": container with ID starting with 9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856 not found: ID does not exist" containerID="9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856"
Oct 30 19:31:42.029022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.029008    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856"} err="failed to get container status \"9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856\": rpc error: code = NotFound desc = could not find container \"9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856\": container with ID starting with 9e8d742d9b499029004aef4b3f519c7194501585414f3036f952765f07b9f856 not found: ID does not exist"
Oct 30 19:31:42.047257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.047219    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"]
Oct 30 19:31:42.051318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.051286    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-pd4lj"]
Oct 30 19:31:42.894061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:42.894023    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:43.264258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:43.264181    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" path="/var/lib/kubelet/pods/8999c9ff-9ed0-4705-93e5-6bca3a32ed68/volumes"
Oct 30 19:31:43.559011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:43.558881    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:44.028909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:44.028881    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"]
Oct 30 19:31:44.032958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:44.032933    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd658e8-x5l8x"]
Oct 30 19:31:44.438964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:44.438932    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:45.265955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:45.265923    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719a9674-bf9f-4f41-89d2-697cedb57e0e" path="/var/lib/kubelet/pods/719a9674-bf9f-4f41-89d2-697cedb57e0e/volumes"
Oct 30 19:31:45.482084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:45.482049    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:46.395832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:46.395804    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:47.067141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.067102    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:47.282550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.282517    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:47.948484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.948453    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:47.960032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.960003    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"]
Oct 30 19:31:47.961679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.961649    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy"
Oct 30 19:31:47.961893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.961877    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy"
Oct 30 19:31:47.962252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.962192    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fa6f2a-2cef-4d7d-8128-58ee4b8107d9" containerName="istio-proxy"
Oct 30 19:31:47.962406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.962393    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8999c9ff-9ed0-4705-93e5-6bca3a32ed68" containerName="istio-proxy"
Oct 30 19:31:47.970267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.970248    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:47.970704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.970663    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"]
Oct 30 19:31:47.977051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.977031    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-02feaa\"/\"openshift-service-ca.crt\""
Oct 30 19:31:47.980802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.980775    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-02feaa\"/\"test-model-02feaa-kserve-self-signed-certs\""
Oct 30 19:31:47.981053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.981038    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-02feaa\"/\"kube-root-ca.crt\""
Oct 30 19:31:47.981319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:47.981304    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-02feaa\"/\"default-dockercfg-tp8vh\""
Oct 30 19:31:48.112415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.112382    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:48.123123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.123098    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-dshm\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.123269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.123131    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-home\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.123269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.123156    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-model-cache\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.123354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.123307    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18e20c5-6289-4bd8-9706-11d6008895ee-tls-certs\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.123354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.123346    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-tmp-dir\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.123461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.123443    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsg8\" (UniqueName: \"kubernetes.io/projected/f18e20c5-6289-4bd8-9706-11d6008895ee-kube-api-access-qrsg8\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.230768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.227017    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-dshm\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.230768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.227065    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-home\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.230768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.227103    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-model-cache\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.230768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.227148    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18e20c5-6289-4bd8-9706-11d6008895ee-tls-certs\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.230768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.227193    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-tmp-dir\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.230768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.227288    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsg8\" (UniqueName: \"kubernetes.io/projected/f18e20c5-6289-4bd8-9706-11d6008895ee-kube-api-access-qrsg8\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.230768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.228473    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-model-cache\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.249142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.249111    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-tmp-dir\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.251720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.251671    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-home\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.265003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.264978    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-dshm\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.271717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.269380    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18e20c5-6289-4bd8-9706-11d6008895ee-tls-certs\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.289715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.286632    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsg8\" (UniqueName: \"kubernetes.io/projected/f18e20c5-6289-4bd8-9706-11d6008895ee-kube-api-access-qrsg8\") pod \"test-model-02feaa-kserve-7944dcdf6f-llwd8\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") " pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.309239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.304030    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:48.671231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.671200    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:48.679997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.679977    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"]
Oct 30 19:31:48.681995 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:31:48.681958    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf18e20c5_6289_4bd8_9706_11d6008895ee.slice/crio-dd7fc670899487b088f509c86492ff0ad12450622a831a3495187d879b8f3494 WatchSource:0}: Error finding container dd7fc670899487b088f509c86492ff0ad12450622a831a3495187d879b8f3494: Status 404 returned error can't find the container with id dd7fc670899487b088f509c86492ff0ad12450622a831a3495187d879b8f3494
Oct 30 19:31:48.873713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:48.872113    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:49.080655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:49.080618    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8" event={"ID":"f18e20c5-6289-4bd8-9706-11d6008895ee","Type":"ContainerStarted","Data":"dd7fc670899487b088f509c86492ff0ad12450622a831a3495187d879b8f3494"}
Oct 30 19:31:49.289455 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:49.289377    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:49.362596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:49.360522    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:50.076449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:50.070900    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:50.106613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:50.106581    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8" event={"ID":"f18e20c5-6289-4bd8-9706-11d6008895ee","Type":"ContainerStarted","Data":"3127655b5f7cb721bba7fccb639f6f78ddd0f136cb23d8e851e14c6a470ed899"}
Oct 30 19:31:50.129543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:50.129490    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8" podStartSLOduration=3.129472309 podStartE2EDuration="3.129472309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:50.124163921 +0000 UTC m=+1545.027176961" watchObservedRunningTime="2026-10-30 19:31:50.129472309 +0000 UTC m=+1545.032485347"
Oct 30 19:31:50.305095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:50.305068    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:50.777936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:50.776794    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:51.771369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:51.771333    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:52.129148 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:52.129063    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:31:52.662713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:52.658641    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:53.428020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:53.424336    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:54.369213 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:54.369176    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:54.537058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:54.535277    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:55.722228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:55.722171    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:56.030415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:56.030314    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"]
Oct 30 19:31:56.048898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:56.048869    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb54977-4vmp9"]
Oct 30 19:31:56.265770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:56.263397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:57.116714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:57.116273    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:57.264703 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:57.264666    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4a8d39-c822-4e84-8c2d-7f57bc070439" path="/var/lib/kubelet/pods/cc4a8d39-c822-4e84-8c2d-7f57bc070439/volumes"
Oct 30 19:31:57.298027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:57.297997    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:57.578337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:57.578300    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:57.999665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:57.999630    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:58.286757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:58.286257    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:58.305350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:58.305316    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:58.321409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:58.321373    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:59.264317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:59.264291    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:31:59.306898 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:31:59.306861    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba76082c_f99e_47f2_8d57_0283bfb7e872.slice/crio-46db9339892077e815ac01292a1c07c7b8d001d201e2e9ea210e74458dce8fff.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a049015_a805_45ec_bbe8_786155bab0bb.slice/crio-3653f5c3d224c803e768e3dc8b1c4e98e652d9d83c785150847780647d685662.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:31:59.517339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:31:59.517243    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:00.200362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:00.200330    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:00.963943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:00.961617    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:01.879417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:01.879383    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:02.476647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:02.476610    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:02.830769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:02.826885    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:03.962709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:03.959896    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:04.809440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:04.809409    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:05.606151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:05.606111    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:05.985346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:05.985312    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"]
Oct 30 19:32:05.990074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:05.990055    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.032641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.032609    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"]
Oct 30 19:32:06.045889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.045856    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d52364\"/\"kube-root-ca.crt\""
Oct 30 19:32:06.046059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.045937    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d52364\"/\"openshift-service-ca.crt\""
Oct 30 19:32:06.049272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.049243    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d52364\"/\"default-dockercfg-kltpt\""
Oct 30 19:32:06.049403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.049338    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d52364\"/\"test-model-d52364-kserve-self-signed-certs\""
Oct 30 19:32:06.049889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.049867    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-dshm\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.050011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.049978    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65lq\" (UniqueName: \"kubernetes.io/projected/8de41fc9-e79a-4141-b3a1-10934169a976-kube-api-access-h65lq\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.050209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.050187    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-tmp-dir\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.050288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.050228    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-home\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.050288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.050252    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-model-cache\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.050409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.050308    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de41fc9-e79a-4141-b3a1-10934169a976-tls-certs\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.151883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.151844    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de41fc9-e79a-4141-b3a1-10934169a976-tls-certs\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.152357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.152331    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-dshm\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.187130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.187093    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65lq\" (UniqueName: \"kubernetes.io/projected/8de41fc9-e79a-4141-b3a1-10934169a976-kube-api-access-h65lq\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.198025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.197990    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-tmp-dir\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.226871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.226840    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-home\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.244770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.227067    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de41fc9-e79a-4141-b3a1-10934169a976-tls-certs\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.244770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.216586    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:06.244770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.227112    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-model-cache\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.244770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.227396    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-model-cache\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.244770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.215737    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-tmp-dir\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.244770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.227626    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-home\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.244770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.216501    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-dshm\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.262437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.262401    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65lq\" (UniqueName: \"kubernetes.io/projected/8de41fc9-e79a-4141-b3a1-10934169a976-kube-api-access-h65lq\") pod \"test-model-d52364-kserve-59db4bcd7f-6c7fh\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") " pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.333799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.311444    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:06.590545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.589761    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:06.684627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:06.684602    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"]
Oct 30 19:32:06.687631 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:32:06.687599    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8de41fc9_e79a_4141_b3a1_10934169a976.slice/crio-0bcf304d0b687fc565647efec6e07b85c00874b78af2406477ca4ba7a4a430ff WatchSource:0}: Error finding container 0bcf304d0b687fc565647efec6e07b85c00874b78af2406477ca4ba7a4a430ff: Status 404 returned error can't find the container with id 0bcf304d0b687fc565647efec6e07b85c00874b78af2406477ca4ba7a4a430ff
Oct 30 19:32:07.362346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:07.362306    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh" event={"ID":"8de41fc9-e79a-4141-b3a1-10934169a976","Type":"ContainerStarted","Data":"b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930"}
Oct 30 19:32:07.362546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:07.362358    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh" event={"ID":"8de41fc9-e79a-4141-b3a1-10934169a976","Type":"ContainerStarted","Data":"0bcf304d0b687fc565647efec6e07b85c00874b78af2406477ca4ba7a4a430ff"}
Oct 30 19:32:07.419756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:07.419708    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh" podStartSLOduration=2.419672782 podStartE2EDuration="2.419672782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:07.417908537 +0000 UTC m=+1562.320921573" watchObservedRunningTime="2026-10-30 19:32:07.419672782 +0000 UTC m=+1562.322685820"
Oct 30 19:32:07.567073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:07.566762    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:08.278949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:08.278918    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:08.330179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:08.330153    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:08.928398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:08.928363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:08.998262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:08.998230    4925 scope.go:122] "RemoveContainer" containerID="061afec097e395fa7bfba1410bf9862d70766d437ba2629dbc9697da9b54440a"
Oct 30 19:32:09.726964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:09.726923    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:10.457572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:10.457531    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:11.177994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:11.177952    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:12.436044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:12.436010    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:13.465326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:13.465292    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:14.821279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:14.821099    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:15.763711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:15.759859    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:16.330037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.330004    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:16.343114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.343091    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:16.452575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.452541    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:16.564410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.564383    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:32:16.737922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.737891    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"]
Oct 30 19:32:16.745156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.745132    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:16.756399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.756367    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"]
Oct 30 19:32:16.758129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.758097    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c86696\"/\"test-model-c86696-kserve-self-signed-certs\""
Oct 30 19:32:16.758280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.758132    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c86696\"/\"kube-root-ca.crt\""
Oct 30 19:32:16.758280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.758206    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c86696\"/\"openshift-service-ca.crt\""
Oct 30 19:32:16.758280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.758259    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c86696\"/\"default-dockercfg-7cgxv\""
Oct 30 19:32:16.833245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.833205    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:16.859541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.859512    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-home\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:16.859730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.859709    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-dshm\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:16.859885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.859865    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-model-cache\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:16.860013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.859998    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qzt\" (UniqueName: \"kubernetes.io/projected/7868c7ae-6114-40b4-9d1c-68e85e127437-kube-api-access-m4qzt\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:16.860077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.860065    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7868c7ae-6114-40b4-9d1c-68e85e127437-tls-certs\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:16.860146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:16.860131    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-tmp-dir\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.011810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.001224    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-dshm\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.011810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.001423    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-model-cache\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.011810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.001599    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qzt\" (UniqueName: \"kubernetes.io/projected/7868c7ae-6114-40b4-9d1c-68e85e127437-kube-api-access-m4qzt\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.011810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.001672    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7868c7ae-6114-40b4-9d1c-68e85e127437-tls-certs\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.011810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.001764    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-tmp-dir\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.011810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.001880    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-home\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.011810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.002499    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-home\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.011810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.002770    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-model-cache\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.047535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.034595    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-tmp-dir\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.062710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.051286    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-dshm\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.062710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.054668    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7868c7ae-6114-40b4-9d1c-68e85e127437-tls-certs\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.103714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.075491    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qzt\" (UniqueName: \"kubernetes.io/projected/7868c7ae-6114-40b4-9d1c-68e85e127437-kube-api-access-m4qzt\") pod \"test-model-c86696-kserve-54f84db5bc-plx5q\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") " pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.370059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.369975    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:17.472387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.472311    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:17.707978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:17.707950    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"]
Oct 30 19:32:17.708373 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:32:17.708349    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7868c7ae_6114_40b4_9d1c_68e85e127437.slice/crio-dc6306fcc0d27b52e8cd87183fd5a9df7279a6e2bf9525f494ceeeeb57fa94ca WatchSource:0}: Error finding container dc6306fcc0d27b52e8cd87183fd5a9df7279a6e2bf9525f494ceeeeb57fa94ca: Status 404 returned error can't find the container with id dc6306fcc0d27b52e8cd87183fd5a9df7279a6e2bf9525f494ceeeeb57fa94ca
Oct 30 19:32:18.271388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:18.271349    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:18.570403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:18.570310    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q" event={"ID":"7868c7ae-6114-40b4-9d1c-68e85e127437","Type":"ContainerStarted","Data":"d384a823f573621c3f99c59838303777c22f33d95828ff4a0ddb0d3564e18b56"}
Oct 30 19:32:18.570403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:18.570361    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q" event={"ID":"7868c7ae-6114-40b4-9d1c-68e85e127437","Type":"ContainerStarted","Data":"dc6306fcc0d27b52e8cd87183fd5a9df7279a6e2bf9525f494ceeeeb57fa94ca"}
Oct 30 19:32:18.590808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:18.590762    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q" podStartSLOduration=2.590748366 podStartE2EDuration="2.590748366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:18.587345357 +0000 UTC m=+1573.490358396" watchObservedRunningTime="2026-10-30 19:32:18.590748366 +0000 UTC m=+1573.493761438"
Oct 30 19:32:19.061727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:19.060633    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:19.370825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:19.370739    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:20.125278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:20.125224    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:20.758420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:20.758354    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:21.550174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:21.550140    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:22.098657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.098625    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"]
Oct 30 19:32:22.099156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.099124    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn" podUID="bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" containerName="maas-api" containerID="cri-o://df4dafa7b8310dfd9603114343bc15b64b22d2d08d4ea67e86a2140ac3c2fac2" gracePeriod=30
Oct 30 19:32:22.167555 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:32:22.167515    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:32:22.202484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.202450    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:22.762076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.761997    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:22.876159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.876132    4925 generic.go:356] "Generic (PLEG): container finished" podID="bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" containerID="df4dafa7b8310dfd9603114343bc15b64b22d2d08d4ea67e86a2140ac3c2fac2" exitCode=0
Oct 30 19:32:22.876531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.876303    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn" event={"ID":"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1","Type":"ContainerDied","Data":"df4dafa7b8310dfd9603114343bc15b64b22d2d08d4ea67e86a2140ac3c2fac2"}
Oct 30 19:32:22.876531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.876465    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn" event={"ID":"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1","Type":"ContainerDied","Data":"f7e4a3cc7cba696a37c4fb051f970ccfd0812d410a93d84215b7743e4825ea9a"}
Oct 30 19:32:22.876531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.876497    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7e4a3cc7cba696a37c4fb051f970ccfd0812d410a93d84215b7743e4825ea9a"
Oct 30 19:32:22.879957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.879679    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:32:22.982047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:22.981971    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:23.020960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.020931    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-openshift-service-ca\") pod \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") "
Oct 30 19:32:23.021158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.021031    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-maas-api-tls\") pod \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") "
Oct 30 19:32:23.021158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.021104    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-kube-api-access-plvhd\" (UniqueName: \"kubernetes.io/projected/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-kube-api-access-plvhd\") pod \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") "
Oct 30 19:32:23.021348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.021302    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-metrics-certs\") pod \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\" (UID: \"bf332f62-5e55-4fc7-8e5b-71fe1129d6c1\") "
Oct 30 19:32:23.021585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.021459    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-openshift-service-ca" pod "bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" (UID: "bf332f62-5e55-4fc7-8e5b-71fe1129d6c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:23.027484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.027417    4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-openshift-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.031859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.031819    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-kube-api-access-plvhd" pod "bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" (UID: "bf332f62-5e55-4fc7-8e5b-71fe1129d6c1"). InnerVolumeSpecName "kube-api-access-plvhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:23.032140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.032115    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-maas-api-tls" pod "bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" (UID: "bf332f62-5e55-4fc7-8e5b-71fe1129d6c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:23.034819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.034795    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-metrics-certs" pod "bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" (UID: "bf332f62-5e55-4fc7-8e5b-71fe1129d6c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:23.129507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.129476    4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-metrics-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.129507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.129501    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-maas-api-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.129507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.129511    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvhd\" (UniqueName: \"kubernetes.io/projected/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1-kube-api-access-plvhd\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:23.139452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.139417    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:23.432528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.432484    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"]
Oct 30 19:32:23.433225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.433205    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" containerName="maas-api"
Oct 30 19:32:23.433225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.433222    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" containerName="maas-api"
Oct 30 19:32:23.433402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.433368    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" containerName="maas-api"
Oct 30 19:32:23.437240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.437220    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.444019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.443998    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-618d93\"/\"auth-test-model-618d93-kserve-self-signed-certs\""
Oct 30 19:32:23.444244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.444224    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-618d93\"/\"default-dockercfg-9nqgt\""
Oct 30 19:32:23.444799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.444773    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"]
Oct 30 19:32:23.451243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.451040    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-618d93\"/\"kube-root-ca.crt\""
Oct 30 19:32:23.451243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.451087    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-618d93\"/\"openshift-service-ca.crt\""
Oct 30 19:32:23.535104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.535012    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-model-cache\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.535104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.535059    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-dshm\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.535349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.535129    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgn6h\" (UniqueName: \"kubernetes.io/projected/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-kube-api-access-fgn6h\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.535349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.535211    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tls-certs\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.535467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.535366    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tmp-dir\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.535467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.535444    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-home\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.545116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.545088    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:23.646721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.646071    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-model-cache\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.646721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.646392    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-dshm\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.646721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.646598    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgn6h\" (UniqueName: \"kubernetes.io/projected/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-kube-api-access-fgn6h\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.647034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.646912    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tls-certs\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.675709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.647973    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tmp-dir\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.675709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.661087    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tmp-dir\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.675709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.669973    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-home\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.675709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.670057    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-model-cache\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.675709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.670437    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-home\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.675709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.675101    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tls-certs\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.713277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.708317    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-dshm\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.766718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.765992    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgn6h\" (UniqueName: \"kubernetes.io/projected/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-kube-api-access-fgn6h\") pod \"auth-test-model-618d93-kserve-689b84985c-5gdlm\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") " pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:23.776887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.773642    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:23.900776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.899368    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"
Oct 30 19:32:23.946452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.944130    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"]
Oct 30 19:32:23.957116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:23.955427    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e6159-768645d7bc-w8cnn"]
Oct 30 19:32:24.056796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:24.056757    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:24.064318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:24.064289    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:24.387237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:24.387210    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"]
Oct 30 19:32:24.388325 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:32:24.388300    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2c102d1_6e7f_4f2f_b00c_c2372298d20a.slice/crio-854cfdbeb6951e4e198192f3eb42aacfc067d61da1ac6059aa7aa97da5843c23 WatchSource:0}: Error finding container 854cfdbeb6951e4e198192f3eb42aacfc067d61da1ac6059aa7aa97da5843c23: Status 404 returned error can't find the container with id 854cfdbeb6951e4e198192f3eb42aacfc067d61da1ac6059aa7aa97da5843c23
Oct 30 19:32:24.590908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:24.590867    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:24.817241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:24.817155    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:24.908988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:24.908950    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm" event={"ID":"e2c102d1-6e7f-4f2f-b00c-c2372298d20a","Type":"ContainerStarted","Data":"b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0"}
Oct 30 19:32:24.909192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:24.909002    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm" event={"ID":"e2c102d1-6e7f-4f2f-b00c-c2372298d20a","Type":"ContainerStarted","Data":"854cfdbeb6951e4e198192f3eb42aacfc067d61da1ac6059aa7aa97da5843c23"}
Oct 30 19:32:24.997383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:24.988619    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm" podStartSLOduration=1.988594773 podStartE2EDuration="1.988594773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:24.942237757 +0000 UTC m=+1579.845250797" watchObservedRunningTime="2026-10-30 19:32:24.988594773 +0000 UTC m=+1579.891607812"
Oct 30 19:32:25.064989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.064959    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:25.372998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.372915    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:25.386794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.375303    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf332f62-5e55-4fc7-8e5b-71fe1129d6c1" path="/var/lib/kubelet/pods/bf332f62-5e55-4fc7-8e5b-71fe1129d6c1/volumes"
Oct 30 19:32:25.594184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.594148    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"]
Oct 30 19:32:25.600995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.599886    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.617090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.616890    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-02feaa-openshift-default-dockercfg-8qgr7\""
Oct 30 19:32:25.629794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.627304    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"]
Oct 30 19:32:25.655304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.655267    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:25.752352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.752318    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03fe9ab-eebf-438a-bc73-d00f216171c6-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.752523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.752406    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.752698 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.752659    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.752756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.752720    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.752817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.752800    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nvv\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-kube-api-access-x5nvv\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.752872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.752848    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.753029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.753013    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.753066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.753052    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.753097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.753075    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855303    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855339    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855374    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nvv\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-kube-api-access-x5nvv\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855404    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855506    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855531    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855559    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855645    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03fe9ab-eebf-438a-bc73-d00f216171c6-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855728    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855780    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.855856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855826    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.856231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.855901    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.856231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.856115    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.856577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.856507    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03fe9ab-eebf-438a-bc73-d00f216171c6-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.858281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.858255    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.858411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.858394    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.883539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.883450    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.886188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.886164    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nvv\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-kube-api-access-x5nvv\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:25.931971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:25.931940    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:26.271699 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:32:26.271656    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03fe9ab_eebf_438a_bc73_d00f216171c6.slice/crio-e89f0b76ffacea537bb9dd9820a9fc5735a01162066170d51080d7151ca3ddbf WatchSource:0}: Error finding container e89f0b76ffacea537bb9dd9820a9fc5735a01162066170d51080d7151ca3ddbf: Status 404 returned error can't find the container with id e89f0b76ffacea537bb9dd9820a9fc5735a01162066170d51080d7151ca3ddbf
Oct 30 19:32:26.271822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:26.271788    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"]
Oct 30 19:32:26.274983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:26.274942    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:26.275089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:26.275035    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:26.275089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:26.275076    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:26.595657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:26.595569    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:26.935381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:26.935329    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" event={"ID":"b03fe9ab-eebf-438a-bc73-d00f216171c6","Type":"ContainerStarted","Data":"bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c"}
Oct 30 19:32:26.935894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:26.935396    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" event={"ID":"b03fe9ab-eebf-438a-bc73-d00f216171c6","Type":"ContainerStarted","Data":"e89f0b76ffacea537bb9dd9820a9fc5735a01162066170d51080d7151ca3ddbf"}
Oct 30 19:32:27.000813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:27.000748    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" podStartSLOduration=2.000731645 podStartE2EDuration="2.000731645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:27.000059984 +0000 UTC m=+1581.903073041" watchObservedRunningTime="2026-10-30 19:32:27.000731645 +0000 UTC m=+1581.903744689"
Oct 30 19:32:27.370607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:27.370516    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:27.383125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:27.383101    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:27.821898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:27.821868    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:27.933161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:27.933129    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:27.935073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:27.935052    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:32:27.935143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:27.935101    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 19:32:27.955122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:27.955096    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:32:28.932569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:28.932531    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:32:28.932797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:28.932586    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 19:32:29.224257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:29.224172    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:29.478716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:29.478609    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:29.718710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:29.718500    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:29.933103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:29.933065    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:32:29.933283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:29.933132    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 19:32:30.639788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:30.636063    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:30.954925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:30.954843    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:32:30.955197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:30.955166    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 19:32:30.967109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:30.967075    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:31.269426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:31.269336    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:31.959273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:31.959232    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:32:31.959944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:31.959909    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 19:32:32.075061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:32.075006    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:32.932928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:32.932898    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 19:32:32.933177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:32.933155    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 19:32:33.222675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:33.222585    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:33.815390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:33.815345    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:33.975027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:33.974318    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:33.975027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:33.974870    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:33.976425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:33.976389    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:32:34.038828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:34.038797    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"]
Oct 30 19:32:34.039060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:34.039036    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy" containerID="cri-o://414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc" gracePeriod=30
Oct 30 19:32:34.057347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:34.057316    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:34.071535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:34.071456    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:34.123434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:34.123403    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:34.357766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:34.357056    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:34.952563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:34.952527    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:35.068567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:35.068538    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:32:35.232412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:35.232301    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:35.479891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:35.479825    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:35.685128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:35.685092    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:36.356464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:36.356358    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:36.416224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:36.416193    4925 patch_prober.go:29] interesting pod/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:32:36.416409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:36.416259    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:32:37.298704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:37.298610    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:38.148202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:38.148170    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:39.326238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.326159    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:39.385448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.385306    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:32:39.582061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.578793    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.583054    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-envoy\") pod \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") "
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.583108    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istiod-ca-cert\") pod \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") "
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.583204    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-socket\") pod \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") "
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.583284    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-data\") pod \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") "
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.583326    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-token\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-token\") pod \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") "
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.583354    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-kube-api-access-bzckb\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-kube-api-access-bzckb\") pod \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") "
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.583475    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-credential-socket\") pod \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") "
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.583666    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-certs\") pod \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") "
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.583719    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-podinfo\") pod \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\" (UID: \"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1\") "
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.588196    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istiod-ca-cert" pod "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" (UID: "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.588584    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-credential-socket" pod "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" (UID: "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.588989    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-data" pod "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" (UID: "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.589247    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-socket" pod "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" (UID: "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:39.589731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.589473    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-certs" pod "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" (UID: "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:39.686056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.685884    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:39.686056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.685918    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:39.686056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.685934    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:39.686056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.685948    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:39.686056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.685962    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:39.720525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.720474    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-envoy" pod "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" (UID: "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:39.725325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.725061    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-token" pod "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" (UID: "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:39.728899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.727918    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-kube-api-access-bzckb" pod "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" (UID: "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1"). InnerVolumeSpecName "kube-api-access-bzckb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:39.735297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.735264    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-podinfo" pod "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" (UID: "8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:32:39.788189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.788152    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:39.788189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.788185    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzckb\" (UniqueName: \"kubernetes.io/projected/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-kube-api-access-bzckb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:39.788189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.788195    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:39.788388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:39.788204    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:40.107455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.107421    4925 generic.go:356] "Generic (PLEG): container finished" podID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerID="414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc" exitCode=0
Oct 30 19:32:40.107653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.107473    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"
Oct 30 19:32:40.107653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.107512    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" event={"ID":"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1","Type":"ContainerDied","Data":"414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc"}
Oct 30 19:32:40.107653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.107612    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c" event={"ID":"8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1","Type":"ContainerDied","Data":"d846d3a433692c134689bdda0687165f0e140658ef49b387e4ddc0db7fc51daf"}
Oct 30 19:32:40.107653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.107641    4925 scope.go:122] "RemoveContainer" containerID="414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc"
Oct 30 19:32:40.120545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.120523    4925 scope.go:122] "RemoveContainer" containerID="22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458"
Oct 30 19:32:40.131408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.131388    4925 scope.go:122] "RemoveContainer" containerID="414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc"
Oct 30 19:32:40.131748 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:32:40.131717    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc\": container with ID starting with 414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc not found: ID does not exist" containerID="414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc"
Oct 30 19:32:40.131837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.131762    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc"} err="failed to get container status \"414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc\": rpc error: code = NotFound desc = could not find container \"414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc\": container with ID starting with 414d46cdec9c671719522bbe2397979d96640bf5b4d7b67b9f3e2087a3d90bfc not found: ID does not exist"
Oct 30 19:32:40.131837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.131789    4925 scope.go:122] "RemoveContainer" containerID="22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458"
Oct 30 19:32:40.132117 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:32:40.132092    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458\": container with ID starting with 22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458 not found: ID does not exist" containerID="22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458"
Oct 30 19:32:40.132201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.132127    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458"} err="failed to get container status \"22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458\": rpc error: code = NotFound desc = could not find container \"22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458\": container with ID starting with 22e8770d2539064eeef6c8db8ff3d392b0faf356f73de2f77fa854e541e61458 not found: ID does not exist"
Oct 30 19:32:40.142675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.142650    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"]
Oct 30 19:32:40.147897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.147874    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8e6159-openshift-default-dd97fcb75-hvm7c"]
Oct 30 19:32:40.810730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:40.804086    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:41.015757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.014768    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:41.099304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.099159    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"]
Oct 30 19:32:41.100071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.100046    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy"
Oct 30 19:32:41.100071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.100069    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy"
Oct 30 19:32:41.100294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.100094    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy"
Oct 30 19:32:41.100294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.100102    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy"
Oct 30 19:32:41.100444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.100428    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy"
Oct 30 19:32:41.100513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.100451    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" containerName="istio-proxy"
Oct 30 19:32:41.105824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.105805    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:41.116753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.116721    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"]
Oct 30 19:32:41.121352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.121328    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f2dbd9\""
Oct 30 19:32:41.135512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.132437    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz\" (UID: \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:41.135512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.132893    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqbq\" (UniqueName: \"kubernetes.io/projected/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-kube-api-access-2tqbq\") pod \"payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz\" (UID: \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:41.167825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.167782    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"]
Oct 30 19:32:41.172909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.172885    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:41.173819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.173791    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"]
Oct 30 19:32:41.177218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.177181    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f2dbd9-dockercfg-54zx8\""
Oct 30 19:32:41.204737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.204710    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:41.234323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.234284    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz\" (UID: \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:41.234483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.234349    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb\" (UID: \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:41.234933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.234905    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6kk2\" (UniqueName: \"kubernetes.io/projected/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-kube-api-access-s6kk2\") pod \"payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb\" (UID: \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:41.235050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.234975    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqbq\" (UniqueName: \"kubernetes.io/projected/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-kube-api-access-2tqbq\") pod \"payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz\" (UID: \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:41.235104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.235056    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz\" (UID: \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:41.249981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.249954    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqbq\" (UniqueName: \"kubernetes.io/projected/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-kube-api-access-2tqbq\") pod \"payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz\" (UID: \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:41.265763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.265734    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1" path="/var/lib/kubelet/pods/8fb1b4ae-9bbe-4202-aeac-28f45cdb7ae1/volumes"
Oct 30 19:32:41.336370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.336337    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6kk2\" (UniqueName: \"kubernetes.io/projected/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-kube-api-access-s6kk2\") pod \"payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb\" (UID: \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:41.336555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.336440    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb\" (UID: \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:41.337166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.337145    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb\" (UID: \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:41.367538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.367469    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6kk2\" (UniqueName: \"kubernetes.io/projected/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-kube-api-access-s6kk2\") pod \"payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb\" (UID: \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:41.469710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.465020    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:41.504856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.503156    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:41.584872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:41.584832    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:42.072468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:42.072434    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"]
Oct 30 19:32:42.072881 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:32:42.072476    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de63bc4_080b_4dc0_9729_f0e2b0fec1ac.slice/crio-35c6c9de6427e5cc6c2b4251d4b269584e059854db511652ed15c403d8986c3e WatchSource:0}: Error finding container 35c6c9de6427e5cc6c2b4251d4b269584e059854db511652ed15c403d8986c3e: Status 404 returned error can't find the container with id 35c6c9de6427e5cc6c2b4251d4b269584e059854db511652ed15c403d8986c3e
Oct 30 19:32:42.079361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:42.079332    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"]
Oct 30 19:32:42.079625 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:32:42.079576    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cd40c7_a0aa_48cc_8e19_e5d626cd6d15.slice/crio-eaeae552b75fad713558bfcb5954c250f515cf4201fec6e8873c33267072e738 WatchSource:0}: Error finding container eaeae552b75fad713558bfcb5954c250f515cf4201fec6e8873c33267072e738: Status 404 returned error can't find the container with id eaeae552b75fad713558bfcb5954c250f515cf4201fec6e8873c33267072e738
Oct 30 19:32:42.154225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:42.154195    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb" event={"ID":"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15","Type":"ContainerStarted","Data":"eaeae552b75fad713558bfcb5954c250f515cf4201fec6e8873c33267072e738"}
Oct 30 19:32:42.155745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:42.155724    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz" event={"ID":"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac","Type":"ContainerStarted","Data":"35c6c9de6427e5cc6c2b4251d4b269584e059854db511652ed15c403d8986c3e"}
Oct 30 19:32:42.392204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:42.392177    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"]
Oct 30 19:32:42.392407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:42.392383    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy" containerID="cri-o://2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d" gracePeriod=30
Oct 30 19:32:42.923316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:42.923279    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:43.165593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:43.165306    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb" event={"ID":"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15","Type":"ContainerStarted","Data":"cd57674fd299a454a42b4eb8edaf0c8857030b03596aa044dac691f1e4905c0c"}
Oct 30 19:32:43.165593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:43.165496    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:43.167186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:43.167161    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz" event={"ID":"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac","Type":"ContainerStarted","Data":"e37adc3b266b78fb899fc98e600928a8120e659bdb3e35a94fb3e63813529c21"}
Oct 30 19:32:43.167297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:43.167234    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:43.193178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:43.193087    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb" podStartSLOduration=2.19306985 podStartE2EDuration="2.19306985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:43.187974097 +0000 UTC m=+1598.090987133" watchObservedRunningTime="2026-10-30 19:32:43.19306985 +0000 UTC m=+1598.096082890"
Oct 30 19:32:43.219231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:43.219185    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz" podStartSLOduration=2.219173032 podStartE2EDuration="2.219173032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:43.216002319 +0000 UTC m=+1598.119015358" watchObservedRunningTime="2026-10-30 19:32:43.219173032 +0000 UTC m=+1598.122186091"
Oct 30 19:32:43.855421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:43.855374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:45.866087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:45.866055    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:46.081198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:46.081167    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:46.549175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:46.549141    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:46.964661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:46.964624    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:47.697802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.695658    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:32:47.763566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.763374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:47.842816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.842783    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-certs\") pod \"a871e328-d34b-40c8-a5b0-55b97d00dd34\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") "
Oct 30 19:32:47.843158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.843140    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-credential-socket\") pod \"a871e328-d34b-40c8-a5b0-55b97d00dd34\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") "
Oct 30 19:32:47.843307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.843292    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-envoy\") pod \"a871e328-d34b-40c8-a5b0-55b97d00dd34\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") "
Oct 30 19:32:47.843802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.843784    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-data\") pod \"a871e328-d34b-40c8-a5b0-55b97d00dd34\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") "
Oct 30 19:32:47.844247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.844223    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-podinfo\") pod \"a871e328-d34b-40c8-a5b0-55b97d00dd34\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") "
Oct 30 19:32:47.844381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.844355    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-token\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-token\") pod \"a871e328-d34b-40c8-a5b0-55b97d00dd34\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") "
Oct 30 19:32:47.844450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.844410    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a871e328-d34b-40c8-a5b0-55b97d00dd34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a871e328-d34b-40c8-a5b0-55b97d00dd34-istiod-ca-cert\") pod \"a871e328-d34b-40c8-a5b0-55b97d00dd34\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") "
Oct 30 19:32:47.844581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.844564    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-kube-api-access-46qq4\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-kube-api-access-46qq4\") pod \"a871e328-d34b-40c8-a5b0-55b97d00dd34\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") "
Oct 30 19:32:47.844744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.844714    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-socket\") pod \"a871e328-d34b-40c8-a5b0-55b97d00dd34\" (UID: \"a871e328-d34b-40c8-a5b0-55b97d00dd34\") "
Oct 30 19:32:47.845638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.843154    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-certs" pod "a871e328-d34b-40c8-a5b0-55b97d00dd34" (UID: "a871e328-d34b-40c8-a5b0-55b97d00dd34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:47.845789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.843372    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-credential-socket" pod "a871e328-d34b-40c8-a5b0-55b97d00dd34" (UID: "a871e328-d34b-40c8-a5b0-55b97d00dd34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:47.845789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.844107    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-data" pod "a871e328-d34b-40c8-a5b0-55b97d00dd34" (UID: "a871e328-d34b-40c8-a5b0-55b97d00dd34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:47.845789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.845768    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-socket" pod "a871e328-d34b-40c8-a5b0-55b97d00dd34" (UID: "a871e328-d34b-40c8-a5b0-55b97d00dd34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:47.846100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.846080    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a871e328-d34b-40c8-a5b0-55b97d00dd34-istiod-ca-cert" pod "a871e328-d34b-40c8-a5b0-55b97d00dd34" (UID: "a871e328-d34b-40c8-a5b0-55b97d00dd34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:47.862677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.861512    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-podinfo" pod "a871e328-d34b-40c8-a5b0-55b97d00dd34" (UID: "a871e328-d34b-40c8-a5b0-55b97d00dd34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:32:47.862677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.861667    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-kube-api-access-46qq4" pod "a871e328-d34b-40c8-a5b0-55b97d00dd34" (UID: "a871e328-d34b-40c8-a5b0-55b97d00dd34"). InnerVolumeSpecName "kube-api-access-46qq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:47.863764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.863682    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-envoy" pod "a871e328-d34b-40c8-a5b0-55b97d00dd34" (UID: "a871e328-d34b-40c8-a5b0-55b97d00dd34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:47.863880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.863736    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-token" pod "a871e328-d34b-40c8-a5b0-55b97d00dd34" (UID: "a871e328-d34b-40c8-a5b0-55b97d00dd34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:47.947499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.947434    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46qq4\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-kube-api-access-46qq4\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:47.947499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.947458    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:47.947499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.947470    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:47.947499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.947479    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:47.947499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.947488    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:47.947499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.947499    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:47.947813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.947512    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:47.947813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.947521    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a871e328-d34b-40c8-a5b0-55b97d00dd34-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:47.947813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:47.947529    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a871e328-d34b-40c8-a5b0-55b97d00dd34-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:48.263970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.263887    4925 generic.go:356] "Generic (PLEG): container finished" podID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerID="2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d" exitCode=0
Oct 30 19:32:48.263970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.263958    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"
Oct 30 19:32:48.263970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.263963    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" event={"ID":"a871e328-d34b-40c8-a5b0-55b97d00dd34","Type":"ContainerDied","Data":"2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d"}
Oct 30 19:32:48.264513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.264021    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548" event={"ID":"a871e328-d34b-40c8-a5b0-55b97d00dd34","Type":"ContainerDied","Data":"1342afec32fa838e6ec18094243901748cf494316d29d329d0bdd7736a709b13"}
Oct 30 19:32:48.264513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.264043    4925 scope.go:122] "RemoveContainer" containerID="2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d"
Oct 30 19:32:48.278012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.277995    4925 scope.go:122] "RemoveContainer" containerID="2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d"
Oct 30 19:32:48.278323 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:32:48.278301    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d\": container with ID starting with 2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d not found: ID does not exist" containerID="2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d"
Oct 30 19:32:48.278390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.278335    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d"} err="failed to get container status \"2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d\": rpc error: code = NotFound desc = could not find container \"2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d\": container with ID starting with 2def48b6cbfd31e6b64f846af19f30e67bd6962ebfcb3d56ff763ad1afba794d not found: ID does not exist"
Oct 30 19:32:48.297782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.297748    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"]
Oct 30 19:32:48.301939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.301918    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-7g548"]
Oct 30 19:32:48.381615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:48.381580    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:49.095551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.095517    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:49.269216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.269173    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" path="/var/lib/kubelet/pods/a871e328-d34b-40c8-a5b0-55b97d00dd34/volumes"
Oct 30 19:32:49.790899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.790868    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"]
Oct 30 19:32:49.791565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.791549    4925 cpu_manager.go:414] "containerMap: removing container" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy"
Oct 30 19:32:49.791565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.791565    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy"
Oct 30 19:32:49.791713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.791703    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a871e328-d34b-40c8-a5b0-55b97d00dd34" containerName="istio-proxy"
Oct 30 19:32:49.796885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.796866    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.865677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.864666    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"]
Oct 30 19:32:49.894080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.894045    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.894449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.894408    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-data\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.894563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.894493    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istiod-ca-cert\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.894710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.894647    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-envoy\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.894822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.894790    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-credential-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.894953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.894876    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-podinfo\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.894953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.894901    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49bhn\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-kube-api-access-49bhn\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.894953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.894926    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-token\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.895204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.895004    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-certs\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.943800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.943763    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:49.996536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.996508    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-podinfo\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.996794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.996545    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49bhn\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-kube-api-access-49bhn\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.996794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.996579    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-token\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.996794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.996668    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-certs\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.996989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.996824    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.996989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.996932    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-data\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.997112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.996995    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istiod-ca-cert\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.997174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.997115    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-envoy\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.997174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.997152    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-certs\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.997274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.997193    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.997274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.997219    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-credential-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.997274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.997237    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-data\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.997551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.997465    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-credential-socket\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.997923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.997903    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istiod-ca-cert\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.999439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.999420    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-envoy\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:49.999544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:49.999526    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-podinfo\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:50.012479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:50.012454    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-token\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:50.012775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:50.012758    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49bhn\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-kube-api-access-49bhn\") pod \"e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") " pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:50.111590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:50.111510    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:50.409847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:50.409822    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"]
Oct 30 19:32:50.413812 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:32:50.413780    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e2058b_e1ac_4eb8_a903_e0a1b1ee45e0.slice/crio-a5c1f555a726fa843f59bebbf121df872ae211dafd04013303d017fc9bf0dbab WatchSource:0}: Error finding container a5c1f555a726fa843f59bebbf121df872ae211dafd04013303d017fc9bf0dbab: Status 404 returned error can't find the container with id a5c1f555a726fa843f59bebbf121df872ae211dafd04013303d017fc9bf0dbab
Oct 30 19:32:50.417253 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:50.417210    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:50.417345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:50.417307    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:50.417385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:50.417372    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:50.794952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:50.794828    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:51.315830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:51.315787    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" event={"ID":"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0","Type":"ContainerStarted","Data":"ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9"}
Oct 30 19:32:51.316164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:51.316128    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" event={"ID":"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0","Type":"ContainerStarted","Data":"a5c1f555a726fa843f59bebbf121df872ae211dafd04013303d017fc9bf0dbab"}
Oct 30 19:32:51.363816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:51.363756    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" podStartSLOduration=2.363739802 podStartE2EDuration="2.363739802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:51.363298071 +0000 UTC m=+1606.266311109" watchObservedRunningTime="2026-10-30 19:32:51.363739802 +0000 UTC m=+1606.266752838"
Oct 30 19:32:51.557022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:51.556991    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:52.120061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:52.112029    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:52.122632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:52.122601    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:32:52.122839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:52.122680    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:32:52.149745 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:32:52.149705    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:32:52.479952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:52.479849    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:53.066124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.066094    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"]
Oct 30 19:32:53.071252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.071229    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.091737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.088939    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"]
Oct 30 19:32:53.121836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.121791    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:32:53.121988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.121852    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:32:53.122508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.122483    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-76117c-serving-cert\""
Oct 30 19:32:53.176205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.176174    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.176369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.176355    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-maas-api-tls\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.176415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.176403    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmpk\" (UniqueName: \"kubernetes.io/projected/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-kube-api-access-bbmpk\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.176547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.176532    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-metrics-certs\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.191663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.191638    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:53.278717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.278055    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-maas-api-tls\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.278717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.278142    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmpk\" (UniqueName: \"kubernetes.io/projected/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-kube-api-access-bbmpk\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.278717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.278307    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-metrics-certs\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.315718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.279720    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.315718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.286956    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.315718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.293900    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-maas-api-tls\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.315718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.315639    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmpk\" (UniqueName: \"kubernetes.io/projected/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-kube-api-access-bbmpk\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.339932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.319654    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-metrics-certs\") pod \"maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.371486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.371459    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:53.422120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.422085    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:53.753240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:53.753209    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"]
Oct 30 19:32:53.764625 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:32:53.763178    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbdf529_2c4d_4fbb_a526_4020fb2cab17.slice/crio-7d0a97c3df14483a9ded29fd2b8a4446ce9a02cf7fd8c2c2cd0ac121a001414b WatchSource:0}: Error finding container 7d0a97c3df14483a9ded29fd2b8a4446ce9a02cf7fd8c2c2cd0ac121a001414b: Status 404 returned error can't find the container with id 7d0a97c3df14483a9ded29fd2b8a4446ce9a02cf7fd8c2c2cd0ac121a001414b
Oct 30 19:32:54.090227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:54.090137    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:54.115075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:54.115040    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:32:54.115330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:54.115305    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:32:54.180932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:54.180896    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:32:54.181178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:54.181166    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:32:54.281391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:54.281364    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:54.433148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:54.433115    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg" event={"ID":"6cbdf529-2c4d-4fbb-a526-4020fb2cab17","Type":"ContainerStarted","Data":"7d0a97c3df14483a9ded29fd2b8a4446ce9a02cf7fd8c2c2cd0ac121a001414b"}
Oct 30 19:32:55.127714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:55.126758    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:32:55.154710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:55.126857    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:32:55.154710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:55.151457    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:55.444207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:55.444159    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg" event={"ID":"6cbdf529-2c4d-4fbb-a526-4020fb2cab17","Type":"ContainerStarted","Data":"7359c64e28a4c4351e989a41c24469551e703a98a0c6d7497008613c89715563"}
Oct 30 19:32:55.444408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:55.444260    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:32:55.471917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:55.471860    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg" podStartSLOduration=1.778977771 podStartE2EDuration="2.471841956s" totalImagesPullingTime="692.864185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:55.468732466 +0000 UTC m=+1610.371745503" watchObservedRunningTime="2026-10-30 19:32:55.471841956 +0000 UTC m=+1610.374854994"
Oct 30 19:32:55.994543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:55.994447    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:56.119856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:56.119810    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:32:56.120051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:56.119875    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:32:56.680845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:56.680811    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:57.112418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.112322    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 30 19:32:57.112418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.112373    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 30 19:32:57.439303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.439274    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"]
Oct 30 19:32:57.444402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.444375    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.467135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.467114    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-69f3a9-openshift-default-dockercfg-fsxb8\""
Oct 30 19:32:57.477883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.477860    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"]
Oct 30 19:32:57.509043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.508898    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-istio-token\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.509359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.509334    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-envoy\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.509570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.509543    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587b6511-ae1d-469e-8ecd-89aede370111-istio-podinfo\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.509704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.509619    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.509704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.509643    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-certs\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.509845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.509726    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587b6511-ae1d-469e-8ecd-89aede370111-istiod-ca-cert\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.509845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.509795    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-credential-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.509845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.509822    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-data\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.510205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.510175    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv24\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-kube-api-access-2kv24\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.612049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.612018    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv24\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-kube-api-access-2kv24\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.612251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.612140    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-istio-token\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.612251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.612185    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-envoy\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.612374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.612254    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587b6511-ae1d-469e-8ecd-89aede370111-istio-podinfo\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.612374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.612320    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.612374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.612347    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-certs\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.612538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.612411    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587b6511-ae1d-469e-8ecd-89aede370111-istiod-ca-cert\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.612538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.612480    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-credential-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.612914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.612886    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-credential-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.613191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.613171    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-certs\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.613343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.613221    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-socket\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.613452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.613328    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-data\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.613590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.613568    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587b6511-ae1d-469e-8ecd-89aede370111-istiod-ca-cert\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.613651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.613612    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-data\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.619627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.619586    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-envoy\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.620097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.619997    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587b6511-ae1d-469e-8ecd-89aede370111-istio-podinfo\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.639910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.639882    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-istio-token\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.642525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.642498    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv24\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-kube-api-access-2kv24\") pod \"e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") " pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.650282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.650240    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:57.762673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.762585    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:57.857142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:57.857062    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:58.066146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.066117    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"]
Oct 30 19:32:58.066987 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:32:58.066964    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587b6511_ae1d_469e_8ecd_89aede370111.slice/crio-5b30e9ab3a226c1a79e2ce6a84e952b4759bd59aaddc8120aaa2a5523d144572 WatchSource:0}: Error finding container 5b30e9ab3a226c1a79e2ce6a84e952b4759bd59aaddc8120aaa2a5523d144572: Status 404 returned error can't find the container with id 5b30e9ab3a226c1a79e2ce6a84e952b4759bd59aaddc8120aaa2a5523d144572
Oct 30 19:32:58.070052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.070015    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:58.070183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.070121    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:58.070183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.070168    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:58.118017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.117985    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:58.118441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.118413    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:58.119162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.119143    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:32:58.473587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.473549    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" event={"ID":"587b6511-ae1d-469e-8ecd-89aede370111","Type":"ContainerStarted","Data":"5b17408889c7c14dd444c9cb628b28ed8c7135f5d55b223030fa23876318446f"}
Oct 30 19:32:58.473776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.473613    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" event={"ID":"587b6511-ae1d-469e-8ecd-89aede370111","Type":"ContainerStarted","Data":"5b30e9ab3a226c1a79e2ce6a84e952b4759bd59aaddc8120aaa2a5523d144572"}
Oct 30 19:32:58.522702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.519072    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" podStartSLOduration=1.5190466200000001 podStartE2EDuration="1.51904662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:58.517096525 +0000 UTC m=+1613.420109562" watchObservedRunningTime="2026-10-30 19:32:58.51904662 +0000 UTC m=+1613.422059706"
Oct 30 19:32:58.540671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.540323    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:58.763491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:58.763366    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:32:59.189205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:59.189173    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:59.764118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:59.764076    4925 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:32:59.764576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:32:59.764126    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:33:00.028841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.027345    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:00.578245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.578217    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"]
Oct 30 19:33:00.583759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.583735    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.613721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.595749    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"]
Oct 30 19:33:00.720544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.720508    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-podinfo\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.720795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.720552    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-data\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.720795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.720710    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-token\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.720922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.720827    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-credential-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.720986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.720972    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.721118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.721100    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istiod-ca-cert\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.721204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.721141    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-certs\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.721282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.721267    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-envoy\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.721323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.721289    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g479j\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-kube-api-access-g479j\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.763559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.763529    4925 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:33:00.763743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.763587    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:33:00.823595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.823549    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.823989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.823747    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istiod-ca-cert\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.823989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.823801    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-certs\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.824096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.823997    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-envoy\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.824096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.824041    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g479j\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-kube-api-access-g479j\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.824173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.824142    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-podinfo\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.824208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.824173    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-data\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.824251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.824236    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-token\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.824339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.824325    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-credential-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.825122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.825095    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-credential-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.825790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.825762    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istiod-ca-cert\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.826062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.826041    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-socket\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.826167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.826138    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-certs\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.826216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.826174    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-data\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.827658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.827635    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-podinfo\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.827960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.827943    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-envoy\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.836814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.836760    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-token\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.837572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.837537    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g479j\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-kube-api-access-g479j\") pod \"e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") " pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:00.890045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.890010    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:00.934072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:00.934043    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:01.280807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.280784    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"]
Oct 30 19:33:01.282097 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:33:01.282073    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fa0ab1_97e7_453d_bcd3_bac5b8f5df48.slice/crio-91b9f4039ea7bc1523827426ce9d95a3d8f56b868a54cf64f2501874c88e455c WatchSource:0}: Error finding container 91b9f4039ea7bc1523827426ce9d95a3d8f56b868a54cf64f2501874c88e455c: Status 404 returned error can't find the container with id 91b9f4039ea7bc1523827426ce9d95a3d8f56b868a54cf64f2501874c88e455c
Oct 30 19:33:01.285657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.285622    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:01.285769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.285714    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:01.285769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.285748    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:01.463919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.463581    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:33:01.549004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.548915    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" event={"ID":"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48","Type":"ContainerStarted","Data":"a2c95115997cb98f1c0da1c64a9a2b72d0e545615fba8e3af0a14ce45dcbdcd0"}
Oct 30 19:33:01.549004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.548980    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" event={"ID":"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48","Type":"ContainerStarted","Data":"91b9f4039ea7bc1523827426ce9d95a3d8f56b868a54cf64f2501874c88e455c"}
Oct 30 19:33:01.586882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.585937    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:01.765712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.763124    4925 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:33:01.765712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.763186    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:33:01.765712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.764165    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:01.935776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:01.935749    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:02.468715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:02.468323    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:02.763775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:02.763083    4925 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:33:02.763775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:02.763158    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:33:02.941715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:02.941624    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:02.941715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:02.941701    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:03.482741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:03.481726    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:03.763290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:03.763204    4925 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:33:03.763290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:03.763266    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:33:03.945714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:03.942278    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:03.945714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:03.942337    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:04.153549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:04.148772    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:04.763246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:04.763213    4925 patch_prober.go:29] interesting pod/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:33:04.763450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:04.763270    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:33:04.915712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:04.913197    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:04.936717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:04.936085    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:04.936717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:04.936156    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:05.767970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:05.767941    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:33:05.768596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:05.768340    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:33:05.769108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:05.769089    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:33:05.822421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:05.822363    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" podStartSLOduration=5.822348331 podStartE2EDuration="5.822348331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:01.584860107 +0000 UTC m=+1616.487873146" watchObservedRunningTime="2026-10-30 19:33:05.822348331 +0000 UTC m=+1620.725361371"
Oct 30 19:33:05.935421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:05.935385    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:05.935729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:05.935679    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:06.054302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:06.042571    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:06.817740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:06.816271    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:06.935236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:06.935196    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:06.935402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:06.935254    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:07.470650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:07.469600    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:07.935542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:07.935510    4925 patch_prober.go:29] interesting pod/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:07.935981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:07.935562    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:08.374829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:08.374730    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:08.940551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:08.940520    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:08.941038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:08.941019    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:08.941747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:08.941729    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:33:09.040378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:09.040348    4925 scope.go:122] "RemoveContainer" containerID="2bda1b613e580a21626c6db5fe31afa8add0fc888502d2db6e262bc49a7a6664"
Oct 30 19:33:09.060796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:09.060772    4925 scope.go:122] "RemoveContainer" containerID="cd9827738de6298261f3e3fa57160bb8eb5419988b098f68f4f5e04e53085d20"
Oct 30 19:33:09.404597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:09.404567    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:10.139071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:10.139025    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:10.980710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:10.979845    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:11.678428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:11.678386    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:12.531327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:12.531296    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:13.199949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:13.199906    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:14.164804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:14.164768    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:15.120317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:15.120281    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:15.463713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:15.460497    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:16.213058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.213016    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:16.508851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.508784    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"]
Oct 30 19:33:16.515286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.515264    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.528824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.528803    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"]
Oct 30 19:33:16.533217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.533183    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c24c01\"/\"openshift-service-ca.crt\""
Oct 30 19:33:16.533417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.533265    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c24c01\"/\"kube-root-ca.crt\""
Oct 30 19:33:16.537418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.533996    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c24c01\"/\"default-dockercfg-fpvb9\""
Oct 30 19:33:16.537418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.534095    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c24c01\"/\"test-model-c24c01-kserve-self-signed-certs\""
Oct 30 19:33:16.537418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.536673    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tmp-dir\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.537418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.536832    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-home\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.537418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.536930    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthqn\" (UniqueName: \"kubernetes.io/projected/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-kube-api-access-gthqn\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.537418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.537107    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tls-certs\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.537418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.537153    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-model-cache\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.537418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.537224    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-dshm\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.639023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.638991    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tmp-dir\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.639198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.639147    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-home\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.639265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.639249    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthqn\" (UniqueName: \"kubernetes.io/projected/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-kube-api-access-gthqn\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.639455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.639438    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tls-certs\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.639549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.639462    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tmp-dir\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.639549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.639482    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-model-cache\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.639657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.639573    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-home\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.639657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.639580    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-dshm\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.639789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.639769    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-model-cache\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.641986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.641955    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-dshm\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.642390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.642369    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tls-certs\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.652605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.652576    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthqn\" (UniqueName: \"kubernetes.io/projected/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-kube-api-access-gthqn\") pod \"test-model-c24c01-kserve-f5554cd8f-d8b8f\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") " pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:16.663361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.663324    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:16.831066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:16.830977    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:17.347443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:17.347414    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"]
Oct 30 19:33:17.348821 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:33:17.348795    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded2fd33e_eed8_40e5_8a3e_7a62db8412c1.slice/crio-7f28bfaf6fb3d3aabf75a712ee339a7e65cd5bd3e359aa8c48808c9701e28b11 WatchSource:0}: Error finding container 7f28bfaf6fb3d3aabf75a712ee339a7e65cd5bd3e359aa8c48808c9701e28b11: Status 404 returned error can't find the container with id 7f28bfaf6fb3d3aabf75a712ee339a7e65cd5bd3e359aa8c48808c9701e28b11
Oct 30 19:33:17.504429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:17.504352    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:18.063125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:18.063080    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f" event={"ID":"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1","Type":"ContainerStarted","Data":"c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd"}
Oct 30 19:33:18.063314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:18.063135    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f" event={"ID":"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1","Type":"ContainerStarted","Data":"7f28bfaf6fb3d3aabf75a712ee339a7e65cd5bd3e359aa8c48808c9701e28b11"}
Oct 30 19:33:18.082865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:18.082815    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f" podStartSLOduration=2.082800804 podStartE2EDuration="2.082800804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:18.08022094 +0000 UTC m=+1632.983233979" watchObservedRunningTime="2026-10-30 19:33:18.082800804 +0000 UTC m=+1632.985813840"
Oct 30 19:33:18.388952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:18.387408    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:18.832093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:18.832065    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:19.224954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:19.224913    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:20.069312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:20.069277    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:20.993897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:20.993849    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:22.060226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:22.060184    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:22.366286 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:33:22.366175    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:33:23.223754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:23.221548    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:24.751464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:24.751431    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:25.892249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:25.892215    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:26.719165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:26.719133    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:26.832227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:26.832197    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:26.845116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:26.845090    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:27.238883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:27.238856    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:33:27.494440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:27.494350    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:28.201627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:28.201595    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:29.047368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:29.047334    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:29.272148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:29.272115    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:29.959048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:29.958896    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:30.637718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:30.636264    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:31.468212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:31.467898    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:32.458777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:32.458675    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:33.203740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.203543    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:33.213749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.208571    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"]
Oct 30 19:33:33.219793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.219767    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.223676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.223316    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"]
Oct 30 19:33:33.226377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.226349    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:33:33.226570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.226546    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:33:33.307094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.307061    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfsrg\" (UniqueName: \"kubernetes.io/projected/3146b621-6068-4071-9154-31fe79722b07-kube-api-access-lfsrg\") pod \"maas-api-revoke-sub-cd8c2e66-m4f4k\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.307306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.307230    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3146b621-6068-4071-9154-31fe79722b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd8c2e66-m4f4k\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.307478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.307459    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3146b621-6068-4071-9154-31fe79722b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd8c2e66-m4f4k\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.409415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.409379    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfsrg\" (UniqueName: \"kubernetes.io/projected/3146b621-6068-4071-9154-31fe79722b07-kube-api-access-lfsrg\") pod \"maas-api-revoke-sub-cd8c2e66-m4f4k\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.409609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.409556    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3146b621-6068-4071-9154-31fe79722b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd8c2e66-m4f4k\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.409804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.409786    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3146b621-6068-4071-9154-31fe79722b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd8c2e66-m4f4k\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.410491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.410388    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3146b621-6068-4071-9154-31fe79722b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd8c2e66-m4f4k\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.412234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.412211    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3146b621-6068-4071-9154-31fe79722b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd8c2e66-m4f4k\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.423334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.423314    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfsrg\" (UniqueName: \"kubernetes.io/projected/3146b621-6068-4071-9154-31fe79722b07-kube-api-access-lfsrg\") pod \"maas-api-revoke-sub-cd8c2e66-m4f4k\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.560141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.560047    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:33.851871 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:33:33.851841    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3146b621_6068_4071_9154_31fe79722b07.slice/crio-86ca19116a500ff59bee36801dc9bc170f9b829bdd17f5bdc31b0eaccc52614b WatchSource:0}: Error finding container 86ca19116a500ff59bee36801dc9bc170f9b829bdd17f5bdc31b0eaccc52614b: Status 404 returned error can't find the container with id 86ca19116a500ff59bee36801dc9bc170f9b829bdd17f5bdc31b0eaccc52614b
Oct 30 19:33:33.852998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:33.852973    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"]
Oct 30 19:33:34.170899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:34.155438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:34.305793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:34.305751    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k" event={"ID":"3146b621-6068-4071-9154-31fe79722b07","Type":"ContainerStarted","Data":"b0b36489436a6d7413ecfc19dbae4811705aecddd0d491b71103684bb599c263"}
Oct 30 19:33:34.305955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:34.305812    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k" event={"ID":"3146b621-6068-4071-9154-31fe79722b07","Type":"ContainerStarted","Data":"86ca19116a500ff59bee36801dc9bc170f9b829bdd17f5bdc31b0eaccc52614b"}
Oct 30 19:33:34.326679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:34.326632    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k" podStartSLOduration=1.326618514 podStartE2EDuration="1.326618514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:34.325076489 +0000 UTC m=+1649.228089525" watchObservedRunningTime="2026-10-30 19:33:34.326618514 +0000 UTC m=+1649.229631572"
Oct 30 19:33:34.965197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:34.965162    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:35.315698 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:35.315655    4925 generic.go:356] "Generic (PLEG): container finished" podID="3146b621-6068-4071-9154-31fe79722b07" containerID="b0b36489436a6d7413ecfc19dbae4811705aecddd0d491b71103684bb599c263" exitCode=0
Oct 30 19:33:35.315877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:35.315752    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k" event={"ID":"3146b621-6068-4071-9154-31fe79722b07","Type":"ContainerDied","Data":"b0b36489436a6d7413ecfc19dbae4811705aecddd0d491b71103684bb599c263"}
Oct 30 19:33:35.644022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:35.643987    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:35.878746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:35.868706    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:36.522611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.522585    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:36.606511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.606429    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3146b621-6068-4071-9154-31fe79722b07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3146b621-6068-4071-9154-31fe79722b07-maas-api-xks-ca\") pod \"3146b621-6068-4071-9154-31fe79722b07\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") "
Oct 30 19:33:36.606681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.606635    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3146b621-6068-4071-9154-31fe79722b07-kube-api-access-lfsrg\" (UniqueName: \"kubernetes.io/projected/3146b621-6068-4071-9154-31fe79722b07-kube-api-access-lfsrg\") pod \"3146b621-6068-4071-9154-31fe79722b07\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") "
Oct 30 19:33:36.606975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.606952    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3146b621-6068-4071-9154-31fe79722b07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3146b621-6068-4071-9154-31fe79722b07-maas-api-openshift-ca\") pod \"3146b621-6068-4071-9154-31fe79722b07\" (UID: \"3146b621-6068-4071-9154-31fe79722b07\") "
Oct 30 19:33:36.607389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.607353    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3146b621-6068-4071-9154-31fe79722b07-maas-api-openshift-ca" pod "3146b621-6068-4071-9154-31fe79722b07" (UID: "3146b621-6068-4071-9154-31fe79722b07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:36.607988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.607914    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3146b621-6068-4071-9154-31fe79722b07-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:36.611754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.610091    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3146b621-6068-4071-9154-31fe79722b07-kube-api-access-lfsrg" pod "3146b621-6068-4071-9154-31fe79722b07" (UID: "3146b621-6068-4071-9154-31fe79722b07"). InnerVolumeSpecName "kube-api-access-lfsrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:36.611754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.610351    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3146b621-6068-4071-9154-31fe79722b07-maas-api-xks-ca" pod "3146b621-6068-4071-9154-31fe79722b07" (UID: "3146b621-6068-4071-9154-31fe79722b07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:36.709504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.709467    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3146b621-6068-4071-9154-31fe79722b07-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:36.709504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.709500    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfsrg\" (UniqueName: \"kubernetes.io/projected/3146b621-6068-4071-9154-31fe79722b07-kube-api-access-lfsrg\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:36.989217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:36.989181    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:37.333889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:37.333857    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k" event={"ID":"3146b621-6068-4071-9154-31fe79722b07","Type":"ContainerDied","Data":"86ca19116a500ff59bee36801dc9bc170f9b829bdd17f5bdc31b0eaccc52614b"}
Oct 30 19:33:37.334055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:37.333910    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ca19116a500ff59bee36801dc9bc170f9b829bdd17f5bdc31b0eaccc52614b"
Oct 30 19:33:37.334055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:37.333872    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"
Oct 30 19:33:37.869509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:37.869475    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:39.011157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:39.011117    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:39.621274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:39.621238    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:40.399745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:40.399714    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:40.985516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:40.985481    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:41.779527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:41.778680    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:42.396124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:42.395496    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:43.177032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:43.177000    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:43.899838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:43.899806    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:44.460410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:44.460374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:44.532663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:44.532622    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"]
Oct 30 19:33:44.639757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:44.639724    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:45.648799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:45.648744    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:46.650100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:46.650062    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:47.641618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:47.641577    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:48.212645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.212596    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:48.468348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.468259    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:48.567586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.567556    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"]
Oct 30 19:33:48.568358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.568343    4925 cpu_manager.go:414] "containerMap: removing container" podUID="3146b621-6068-4071-9154-31fe79722b07" containerName="revoke-keys"
Oct 30 19:33:48.568406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.568361    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3146b621-6068-4071-9154-31fe79722b07" containerName="revoke-keys"
Oct 30 19:33:48.568489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.568480    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3146b621-6068-4071-9154-31fe79722b07" containerName="revoke-keys"
Oct 30 19:33:48.573363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.573343    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.577061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.577039    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:33:48.577573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.577534    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:33:48.581722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.581660    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"]
Oct 30 19:33:48.678475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.678443    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-995a9ec3-tlfvn\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.678640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.678500    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-995a9ec3-tlfvn\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.678925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.678903    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtg5\" (UniqueName: \"kubernetes.io/projected/028c7f0d-a473-4753-8096-65af4f9264b5-kube-api-access-crtg5\") pod \"maas-api-revoke-sub-995a9ec3-tlfvn\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.781314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.781229    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crtg5\" (UniqueName: \"kubernetes.io/projected/028c7f0d-a473-4753-8096-65af4f9264b5-kube-api-access-crtg5\") pod \"maas-api-revoke-sub-995a9ec3-tlfvn\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.781470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.781377    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-995a9ec3-tlfvn\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.781470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.781410    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-995a9ec3-tlfvn\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.782207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.782181    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-995a9ec3-tlfvn\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.784054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.784034    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-995a9ec3-tlfvn\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.793604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.793578    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtg5\" (UniqueName: \"kubernetes.io/projected/028c7f0d-a473-4753-8096-65af4f9264b5-kube-api-access-crtg5\") pod \"maas-api-revoke-sub-995a9ec3-tlfvn\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:48.892682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:48.892654    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:49.180161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:49.180135    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"]
Oct 30 19:33:49.266711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:49.262515    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:49.486541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:49.486445    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn" event={"ID":"028c7f0d-a473-4753-8096-65af4f9264b5","Type":"ContainerStarted","Data":"78d8fb4391aca91c056ea6f028330e8889d3fca9d134203ef1dc9601e8d8fac8"}
Oct 30 19:33:49.931962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:49.931918    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:50.497778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:50.497654    4925 generic.go:356] "Generic (PLEG): container finished" podID="028c7f0d-a473-4753-8096-65af4f9264b5" containerID="f1a4fa788a0bf531d508ff6455427b2e0ce37e23530f43c417c8acf8808c51f0" exitCode=0
Oct 30 19:33:50.498144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:50.497769    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn" event={"ID":"028c7f0d-a473-4753-8096-65af4f9264b5","Type":"ContainerDied","Data":"f1a4fa788a0bf531d508ff6455427b2e0ce37e23530f43c417c8acf8808c51f0"}
Oct 30 19:33:50.623979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:50.623951    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:51.032612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.032578    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"]
Oct 30 19:33:51.039084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.039059    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692a3083-drsvq"]
Oct 30 19:33:51.265193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.265157    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74313a0-412d-4e3e-98f0-1bbccf4fa459" path="/var/lib/kubelet/pods/a74313a0-412d-4e3e-98f0-1bbccf4fa459/volumes"
Oct 30 19:33:51.723118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.723087    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:51.750955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.750918    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:51.765555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.765479    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-openshift-ca\") pod \"028c7f0d-a473-4753-8096-65af4f9264b5\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") "
Oct 30 19:33:51.765733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.765668    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-xks-ca\") pod \"028c7f0d-a473-4753-8096-65af4f9264b5\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") "
Oct 30 19:33:51.765863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.765840    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-openshift-ca" pod "028c7f0d-a473-4753-8096-65af4f9264b5" (UID: "028c7f0d-a473-4753-8096-65af4f9264b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:51.765961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.765851    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028c7f0d-a473-4753-8096-65af4f9264b5-kube-api-access-crtg5\" (UniqueName: \"kubernetes.io/projected/028c7f0d-a473-4753-8096-65af4f9264b5-kube-api-access-crtg5\") pod \"028c7f0d-a473-4753-8096-65af4f9264b5\" (UID: \"028c7f0d-a473-4753-8096-65af4f9264b5\") "
Oct 30 19:33:51.766606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.766589    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:51.768029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.768006    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-xks-ca" pod "028c7f0d-a473-4753-8096-65af4f9264b5" (UID: "028c7f0d-a473-4753-8096-65af4f9264b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:51.768107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.768033    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028c7f0d-a473-4753-8096-65af4f9264b5-kube-api-access-crtg5" pod "028c7f0d-a473-4753-8096-65af4f9264b5" (UID: "028c7f0d-a473-4753-8096-65af4f9264b5"). InnerVolumeSpecName "kube-api-access-crtg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:51.868926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.868891    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crtg5\" (UniqueName: \"kubernetes.io/projected/028c7f0d-a473-4753-8096-65af4f9264b5-kube-api-access-crtg5\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:51.868926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:51.868920    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028c7f0d-a473-4753-8096-65af4f9264b5-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.129452 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:33:52.129371    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:33:52.566716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:52.566017    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn" event={"ID":"028c7f0d-a473-4753-8096-65af4f9264b5","Type":"ContainerDied","Data":"78d8fb4391aca91c056ea6f028330e8889d3fca9d134203ef1dc9601e8d8fac8"}
Oct 30 19:33:52.566716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:52.566077    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d8fb4391aca91c056ea6f028330e8889d3fca9d134203ef1dc9601e8d8fac8"
Oct 30 19:33:52.566716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:52.566144    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"
Oct 30 19:33:53.234028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.233978    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f" podUID="ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.164:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.250844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.250787    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8" podUID="f18e20c5-6289-4bd8-9706-11d6008895ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.153:8000/ready\": context deadline exceeded"
Oct 30 19:33:53.252106 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.252074    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.359717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.358723    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.413857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.413812    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.485899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.485781    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.490097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.490061    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:33:53.569615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.569578    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh" podUID="8de41fc9-e79a-4141-b3a1-10934169a976" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.570245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.570218    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp" podUID="68f400a1-258e-4e0e-99ea-a440a7c12939" containerName="main" probeResult="failure" output="Get \"https://10.132.0.150:8000/ready\": context deadline exceeded"
Oct 30 19:33:53.572036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.572014    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:53.604854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.599196    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.605210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.599245    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn" podUID="3f1076a2-62c1-4765-b1f0-6465fc85c599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.152:8000/ready\": context deadline exceeded"
Oct 30 19:33:53.605370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.599290    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:33:53.671716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.664855    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.702757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.701855    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.728712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.728570    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.760769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.759913    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:53.801729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:53.797537    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:33:54.277167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:54.277132    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:55.028453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:55.028412    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:55.234121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:55.232606    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:55.651798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:55.651721    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:56.237144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:56.237112    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:56.906128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:56.906089    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:57.446633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:57.446598    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:57.605655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:57.605539    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:58.443741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:58.443708    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:59.114487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.114451    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:59.567922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.567865    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn" podUID="3f1076a2-62c1-4765-b1f0-6465fc85c599" containerName="main" containerID="cri-o://01bc6a21766a80dff75f8f01c2cbe1260474d6220013bad861899eea5310eb66" gracePeriod=45
Oct 30 19:33:59.582198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.582160    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn" podUID="3f1076a2-62c1-4765-b1f0-6465fc85c599" containerName="main" probeResult="failure" output="Get \"https://10.132.0.152:8000/ready\": dial tcp 10.132.0.152:8000: connect: connection refused"
Oct 30 19:33:59.716084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.716056    4925 generic.go:356] "Generic (PLEG): container finished" podID="3f1076a2-62c1-4765-b1f0-6465fc85c599" containerID="01bc6a21766a80dff75f8f01c2cbe1260474d6220013bad861899eea5310eb66" exitCode=0
Oct 30 19:33:59.716218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.716123    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn" event={"ID":"3f1076a2-62c1-4765-b1f0-6465fc85c599","Type":"ContainerDied","Data":"01bc6a21766a80dff75f8f01c2cbe1260474d6220013bad861899eea5310eb66"}
Oct 30 19:33:59.883500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.883471    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:33:59.923952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.923919    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-home\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-home\") pod \"3f1076a2-62c1-4765-b1f0-6465fc85c599\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") "
Oct 30 19:33:59.924136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.923971    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1076a2-62c1-4765-b1f0-6465fc85c599-kube-api-access-9tlm8\" (UniqueName: \"kubernetes.io/projected/3f1076a2-62c1-4765-b1f0-6465fc85c599-kube-api-access-9tlm8\") pod \"3f1076a2-62c1-4765-b1f0-6465fc85c599\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") "
Oct 30 19:33:59.924212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.924167    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-home" pod "3f1076a2-62c1-4765-b1f0-6465fc85c599" (UID: "3f1076a2-62c1-4765-b1f0-6465fc85c599"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:59.924273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.924246    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-model-cache\") pod \"3f1076a2-62c1-4765-b1f0-6465fc85c599\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") "
Oct 30 19:33:59.924331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.924311    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1076a2-62c1-4765-b1f0-6465fc85c599-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1076a2-62c1-4765-b1f0-6465fc85c599-tls-certs\") pod \"3f1076a2-62c1-4765-b1f0-6465fc85c599\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") "
Oct 30 19:33:59.924395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.924360    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-tmp-dir\") pod \"3f1076a2-62c1-4765-b1f0-6465fc85c599\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") "
Oct 30 19:33:59.924451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.924396    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-dshm\") pod \"3f1076a2-62c1-4765-b1f0-6465fc85c599\" (UID: \"3f1076a2-62c1-4765-b1f0-6465fc85c599\") "
Oct 30 19:33:59.924498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.924482    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-model-cache" pod "3f1076a2-62c1-4765-b1f0-6465fc85c599" (UID: "3f1076a2-62c1-4765-b1f0-6465fc85c599"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:59.924637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.924616    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-tmp-dir" pod "3f1076a2-62c1-4765-b1f0-6465fc85c599" (UID: "3f1076a2-62c1-4765-b1f0-6465fc85c599"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:59.925511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.925491    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:59.925629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.925535    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:59.925629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.925549    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:59.926747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.926712    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1076a2-62c1-4765-b1f0-6465fc85c599-tls-certs" pod "3f1076a2-62c1-4765-b1f0-6465fc85c599" (UID: "3f1076a2-62c1-4765-b1f0-6465fc85c599"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:59.927088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.927070    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1076a2-62c1-4765-b1f0-6465fc85c599-kube-api-access-9tlm8" pod "3f1076a2-62c1-4765-b1f0-6465fc85c599" (UID: "3f1076a2-62c1-4765-b1f0-6465fc85c599"). InnerVolumeSpecName "kube-api-access-9tlm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:59.927174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:33:59.927163    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-dshm" pod "3f1076a2-62c1-4765-b1f0-6465fc85c599" (UID: "3f1076a2-62c1-4765-b1f0-6465fc85c599"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:00.035446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.035410    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tlm8\" (UniqueName: \"kubernetes.io/projected/3f1076a2-62c1-4765-b1f0-6465fc85c599-kube-api-access-9tlm8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:00.035742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.035715    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1076a2-62c1-4765-b1f0-6465fc85c599-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:00.035742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.035742    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1076a2-62c1-4765-b1f0-6465fc85c599-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:00.079457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.079426    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:00.795004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.794969    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:00.800243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.800194    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn" event={"ID":"3f1076a2-62c1-4765-b1f0-6465fc85c599","Type":"ContainerDied","Data":"a04dc18b84c50eab753ede345083f9b2f38ea86db383a21002f1a99db96c6fb8"}
Oct 30 19:34:00.800413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.800253    4925 scope.go:122] "RemoveContainer" containerID="01bc6a21766a80dff75f8f01c2cbe1260474d6220013bad861899eea5310eb66"
Oct 30 19:34:00.800484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.800437    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"
Oct 30 19:34:00.833757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.833652    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"]
Oct 30 19:34:00.839298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:00.839272    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ef5dca/test-model-ef5dca-kserve-7696c4f855-98htn"]
Oct 30 19:34:01.266796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:01.266761    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1076a2-62c1-4765-b1f0-6465fc85c599" path="/var/lib/kubelet/pods/3f1076a2-62c1-4765-b1f0-6465fc85c599/volumes"
Oct 30 19:34:01.396832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:01.396789    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:01.881875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:01.881832    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"]
Oct 30 19:34:01.884959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:01.884880    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp" podUID="bde62c69-0252-4744-9d52-85185abad07d" containerName="maas-api" containerID="cri-o://d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0" gracePeriod=30
Oct 30 19:34:01.987076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:01.987038    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:02.183270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.179356    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:02.557317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.541034    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:02.631372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.631345    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:34:02.641505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.641473    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-maas-api-tls\") pod \"bde62c69-0252-4744-9d52-85185abad07d\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") "
Oct 30 19:34:02.641661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.641525    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde62c69-0252-4744-9d52-85185abad07d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde62c69-0252-4744-9d52-85185abad07d-openshift-service-ca\") pod \"bde62c69-0252-4744-9d52-85185abad07d\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") "
Oct 30 19:34:02.641661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.641632    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-metrics-certs\") pod \"bde62c69-0252-4744-9d52-85185abad07d\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") "
Oct 30 19:34:02.641809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.641722    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde62c69-0252-4744-9d52-85185abad07d-kube-api-access-bhz7r\" (UniqueName: \"kubernetes.io/projected/bde62c69-0252-4744-9d52-85185abad07d-kube-api-access-bhz7r\") pod \"bde62c69-0252-4744-9d52-85185abad07d\" (UID: \"bde62c69-0252-4744-9d52-85185abad07d\") "
Oct 30 19:34:02.641952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.641928    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde62c69-0252-4744-9d52-85185abad07d-openshift-service-ca" pod "bde62c69-0252-4744-9d52-85185abad07d" (UID: "bde62c69-0252-4744-9d52-85185abad07d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:02.642482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.642458    4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde62c69-0252-4744-9d52-85185abad07d-openshift-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:02.644174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.644145    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-maas-api-tls" pod "bde62c69-0252-4744-9d52-85185abad07d" (UID: "bde62c69-0252-4744-9d52-85185abad07d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:02.644271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.644224    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-metrics-certs" pod "bde62c69-0252-4744-9d52-85185abad07d" (UID: "bde62c69-0252-4744-9d52-85185abad07d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:02.645099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.645077    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde62c69-0252-4744-9d52-85185abad07d-kube-api-access-bhz7r" pod "bde62c69-0252-4744-9d52-85185abad07d" (UID: "bde62c69-0252-4744-9d52-85185abad07d"). InnerVolumeSpecName "kube-api-access-bhz7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:02.677818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.677783    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"]
Oct 30 19:34:02.678542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.678520    4925 cpu_manager.go:414] "containerMap: removing container" podUID="028c7f0d-a473-4753-8096-65af4f9264b5" containerName="revoke-keys"
Oct 30 19:34:02.678542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.678536    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028c7f0d-a473-4753-8096-65af4f9264b5" containerName="revoke-keys"
Oct 30 19:34:02.678760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.678549    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bde62c69-0252-4744-9d52-85185abad07d" containerName="maas-api"
Oct 30 19:34:02.678760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.678554    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde62c69-0252-4744-9d52-85185abad07d" containerName="maas-api"
Oct 30 19:34:02.678760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.678565    4925 cpu_manager.go:414] "containerMap: removing container" podUID="3f1076a2-62c1-4765-b1f0-6465fc85c599" containerName="main"
Oct 30 19:34:02.678760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.678570    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1076a2-62c1-4765-b1f0-6465fc85c599" containerName="main"
Oct 30 19:34:02.678760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.678703    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="028c7f0d-a473-4753-8096-65af4f9264b5" containerName="revoke-keys"
Oct 30 19:34:02.678760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.678716    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde62c69-0252-4744-9d52-85185abad07d" containerName="maas-api"
Oct 30 19:34:02.678760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.678726    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1076a2-62c1-4765-b1f0-6465fc85c599" containerName="main"
Oct 30 19:34:02.682699 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.682658    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.689449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.689428    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e3586f-openshift-default-dockercfg-czzb7\""
Oct 30 19:34:02.700753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.700726    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"]
Oct 30 19:34:02.741785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.741747    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:02.744309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.744283    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.744437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.744420    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.744620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.744601    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.744744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.744628    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.744744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.744663    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.744744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.744716    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.744888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.744771    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwf4f\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-kube-api-access-pwf4f\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.744888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.744848    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.744888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.744863    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d958656-4816-478b-b8e0-ffcd59d21fc2-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.745065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.745041    4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-metrics-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:02.745181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.745071    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhz7r\" (UniqueName: \"kubernetes.io/projected/bde62c69-0252-4744-9d52-85185abad07d-kube-api-access-bhz7r\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:02.745181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.745087    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde62c69-0252-4744-9d52-85185abad07d-maas-api-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:02.827419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.827317    4925 generic.go:356] "Generic (PLEG): container finished" podID="bde62c69-0252-4744-9d52-85185abad07d" containerID="d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0" exitCode=0
Oct 30 19:34:02.827419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.827379    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"
Oct 30 19:34:02.827419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.827386    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp" event={"ID":"bde62c69-0252-4744-9d52-85185abad07d","Type":"ContainerDied","Data":"d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0"}
Oct 30 19:34:02.827706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.827488    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp" event={"ID":"bde62c69-0252-4744-9d52-85185abad07d","Type":"ContainerDied","Data":"3823e24b93b4e976f583b182c9b2a6ea0b78c134cba40f860ed099311582539d"}
Oct 30 19:34:02.827706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.827527    4925 scope.go:122] "RemoveContainer" containerID="d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0"
Oct 30 19:34:02.846644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.846372    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.846644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.846525    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.847056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.846733    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.847056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.846775    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.847056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.846811    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.847056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.846838    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.847056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.846923    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwf4f\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-kube-api-access-pwf4f\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.847056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.846996    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.847056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.847018    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d958656-4816-478b-b8e0-ffcd59d21fc2-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.847972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.847941    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d958656-4816-478b-b8e0-ffcd59d21fc2-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.848954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.848930    4925 scope.go:122] "RemoveContainer" containerID="d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0"
Oct 30 19:34:02.849603 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:02.849575    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0\": container with ID starting with d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0 not found: ID does not exist" containerID="d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0"
Oct 30 19:34:02.849768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.849615    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0"} err="failed to get container status \"d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0\": rpc error: code = NotFound desc = could not find container \"d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0\": container with ID starting with d4aa9da463f33ee5b1214c4c29d652f0e699c70e8c407f6ded4f86624ebcc9a0 not found: ID does not exist"
Oct 30 19:34:02.850547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.850523    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.851995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.850661    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.851995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.851170    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.852307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.852283    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.852508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.852487    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.853075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.853053    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.879050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.879000    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.886056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.881450    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwf4f\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-kube-api-access-pwf4f\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:02.886955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.886929    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"]
Oct 30 19:34:02.891517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.891492    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-891d80-6f74b64499-tnxbp"]
Oct 30 19:34:02.927003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:02.926972    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:03.006225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.006183    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:03.155656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.155620    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"]
Oct 30 19:34:03.267750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.261398    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:03.310710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.310001    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde62c69-0252-4744-9d52-85185abad07d" path="/var/lib/kubelet/pods/bde62c69-0252-4744-9d52-85185abad07d/volumes"
Oct 30 19:34:03.488566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.486300    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"]
Oct 30 19:34:03.514176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.513670    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:03.514176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.513877    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:03.514176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.513947    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:03.839294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.839202    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" event={"ID":"8d958656-4816-478b-b8e0-ffcd59d21fc2","Type":"ContainerStarted","Data":"6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557"}
Oct 30 19:34:03.839294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.839265    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" event={"ID":"8d958656-4816-478b-b8e0-ffcd59d21fc2","Type":"ContainerStarted","Data":"f439fecbda4e08ff87890ed44fdc332cf78f74124cd8ee0c1ba597a9e3ac22ed"}
Oct 30 19:34:03.872700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:03.872637    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" podStartSLOduration=17.872622665 podStartE2EDuration="17.872622665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:03.869292735 +0000 UTC m=+1678.772305773" watchObservedRunningTime="2026-10-30 19:34:03.872622665 +0000 UTC m=+1678.775635703"
Oct 30 19:34:04.022393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:04.011283    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:04.041716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:04.028827    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:34:04.041716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:04.028912    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 19:34:04.059603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:04.059127    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:05.007711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:05.006764    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:34:05.007711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:05.006840    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 19:34:05.289021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:05.288940    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:05.892568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:05.892534    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:06.015713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:06.010022    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:34:06.015713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:06.010079    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 19:34:06.573402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:06.570165    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:07.013714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:07.006993    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:34:07.013714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:07.007050    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 19:34:07.460252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:07.460219    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:08.019723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:08.019088    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:34:08.019723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:08.019155    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 19:34:08.139721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:08.136407    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:08.431397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:08.431363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:08.771765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:08.769944    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:09.018713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:09.018059    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:34:09.018713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:09.018139    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 19:34:09.122633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:09.122546    4925 scope.go:122] "RemoveContainer" containerID="f76764b02cc12ef4abfd13f1e6256b034c9ee3a2f732287ffa65f12abacdb391"
Oct 30 19:34:09.678721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:09.678335    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:10.006832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:10.006743    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 19:34:10.006832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:10.006808    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 19:34:10.264798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:10.264711    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:10.436741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:10.435872    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:10.619504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:10.619422    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"]
Oct 30 19:34:10.619665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:10.619614    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy" containerID="cri-o://ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9" gracePeriod=30
Oct 30 19:34:10.632448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:10.632419    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"]
Oct 30 19:34:10.632616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:10.632598    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy" containerID="cri-o://a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920" gracePeriod=30
Oct 30 19:34:10.762822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:10.762773    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:11.047097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:11.047057    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:11.048290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:11.048264    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:11.050134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:11.050116    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:34:11.721418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:11.721379    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:12.575436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:12.575331    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:13.121223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:13.121187    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:34:13.121622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:13.121254    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:34:13.350300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:13.350213    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:13.727736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:13.727144    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:13.940358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:13.940327    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:14.347828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:14.347788    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:14.711520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:14.711487    4925 patch_prober.go:29] interesting pod/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:34:14.711744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:14.711548    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:34:15.146961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:15.146922    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:15.974603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:15.974545    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:15.984957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:15.984627    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:34:16.035707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.033560    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"]
Oct 30 19:34:16.035707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.034870    4925 cpu_manager.go:414] "containerMap: removing container" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy"
Oct 30 19:34:16.035707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.034893    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy"
Oct 30 19:34:16.035707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.035156    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerName="istio-proxy"
Oct 30 19:34:16.045939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.042737    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.060218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.058634    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"]
Oct 30 19:34:16.089161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089134    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-certs\") pod \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") "
Oct 30 19:34:16.089341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089181    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-kube-api-access-49bhn\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-kube-api-access-49bhn\") pod \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") "
Oct 30 19:34:16.089341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089314    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-token\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-token\") pod \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") "
Oct 30 19:34:16.089472 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089455    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-socket\") pod \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") "
Oct 30 19:34:16.089527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089479    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-certs" pod "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" (UID: "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.089673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089626    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-credential-socket\") pod \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") "
Oct 30 19:34:16.089763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089719    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-socket" pod "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" (UID: "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.089763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089738    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-envoy\") pod \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") "
Oct 30 19:34:16.089893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089835    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istiod-ca-cert\") pod \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") "
Oct 30 19:34:16.089893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089871    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-podinfo\") pod \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") "
Oct 30 19:34:16.090013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089906    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-credential-socket" pod "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" (UID: "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.090013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.089919    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-data\") pod \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\" (UID: \"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0\") "
Oct 30 19:34:16.090202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.090181    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istiod-ca-cert" pod "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" (UID: "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:16.090367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.090350    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-data" pod "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" (UID: "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.091048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.091022    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.091165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.091052    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.091165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.091067    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.091165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.091080    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.091165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.091098    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.093534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.093501    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-token" pod "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" (UID: "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:16.093782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.093760    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-kube-api-access-49bhn" pod "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" (UID: "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0"). InnerVolumeSpecName "kube-api-access-49bhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:16.094545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.094521    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-envoy" pod "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" (UID: "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.095143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.095117    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-podinfo" pod "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" (UID: "91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:16.106733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.106670    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:34:16.152551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.152522    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"]
Oct 30 19:34:16.153300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.153286    4925 cpu_manager.go:414] "containerMap: removing container" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy"
Oct 30 19:34:16.153371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.153303    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy"
Oct 30 19:34:16.153371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.153333    4925 cpu_manager.go:414] "containerMap: removing container" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy"
Oct 30 19:34:16.153371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.153338    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy"
Oct 30 19:34:16.153494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.153469    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy"
Oct 30 19:34:16.153545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.153501    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerName="istio-proxy"
Oct 30 19:34:16.157473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.157450    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.159538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.159503    4925 generic.go:356] "Generic (PLEG): container finished" podID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" containerID="a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920" exitCode=0
Oct 30 19:34:16.159738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.159598    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" event={"ID":"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd","Type":"ContainerDied","Data":"a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920"}
Oct 30 19:34:16.159738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.159642    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592" event={"ID":"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd","Type":"ContainerDied","Data":"5f42dc163235e3ef91d46605eac4e23a43c6f5dd3eabe599f4ebaf1fa17d816c"}
Oct 30 19:34:16.159738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.159676    4925 scope.go:122] "RemoveContainer" containerID="a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920"
Oct 30 19:34:16.165160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.159891    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"
Oct 30 19:34:16.165160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.163092    4925 generic.go:356] "Generic (PLEG): container finished" podID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" containerID="ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9" exitCode=0
Oct 30 19:34:16.165160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.163432    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" event={"ID":"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0","Type":"ContainerDied","Data":"ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9"}
Oct 30 19:34:16.165160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.163485    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2" event={"ID":"91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0","Type":"ContainerDied","Data":"a5c1f555a726fa843f59bebbf121df872ae211dafd04013303d017fc9bf0dbab"}
Oct 30 19:34:16.165160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.163568    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"
Oct 30 19:34:16.166076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.166054    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f2dbd9-serving-cert\""
Oct 30 19:34:16.167096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.167058    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"]
Oct 30 19:34:16.179775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.178271    4925 scope.go:122] "RemoveContainer" containerID="a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d"
Oct 30 19:34:16.192216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.192191    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-kube-api-access-cwksh\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-kube-api-access-cwksh\") pod \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") "
Oct 30 19:34:16.192355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.192340    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-token\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-token\") pod \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") "
Oct 30 19:34:16.192446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.192434    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-data\") pod \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") "
Oct 30 19:34:16.192535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.192522    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-podinfo\") pod \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") "
Oct 30 19:34:16.192850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.192605    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-credential-socket\") pod \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") "
Oct 30 19:34:16.192850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.192661    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-socket\") pod \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") "
Oct 30 19:34:16.192850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.192757    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istiod-ca-cert\") pod \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") "
Oct 30 19:34:16.192850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.192779    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-certs\") pod \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") "
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.194643    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-socket" pod "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" (UID: "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.194794    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-envoy\") pod \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\" (UID: \"4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd\") "
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.194813    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-credential-socket" pod "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" (UID: "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.194982    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-data" pod "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" (UID: "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195174    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195235    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195289    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715601fa-f650-42e8-a545-ab33d4f3179c-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195299    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istiod-ca-cert" pod "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" (UID: "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195313    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715601fa-f650-42e8-a545-ab33d4f3179c-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195436    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195461    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kl8g\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-kube-api-access-9kl8g\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195486    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.195830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195554    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.196379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.195909    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.196379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.196142    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.196379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.196163    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.196379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.196179    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.196379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.196195    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.196379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.196232    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.196379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.196248    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.196379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.196265    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49bhn\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-kube-api-access-49bhn\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.196379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.196281    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.196816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.196563    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-certs" pod "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" (UID: "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.197042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.197005    4925 scope.go:122] "RemoveContainer" containerID="a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920"
Oct 30 19:34:16.197454 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:16.197296    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920\": container with ID starting with a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920 not found: ID does not exist" containerID="a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920"
Oct 30 19:34:16.197454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.197337    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920"} err="failed to get container status \"a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920\": rpc error: code = NotFound desc = could not find container \"a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920\": container with ID starting with a6aeb660d3394da848db78318608d30f85efa57b6590e3c4c2466470cba8f920 not found: ID does not exist"
Oct 30 19:34:16.197454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.197361    4925 scope.go:122] "RemoveContainer" containerID="a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d"
Oct 30 19:34:16.197775 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:16.197682    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d\": container with ID starting with a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d not found: ID does not exist" containerID="a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d"
Oct 30 19:34:16.197862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.197786    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d"} err="failed to get container status \"a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d\": rpc error: code = NotFound desc = could not find container \"a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d\": container with ID starting with a19fb7474661a32b4cc2b95f9822154cd2dbf74258ffebfa7aec50bb5a2df97d not found: ID does not exist"
Oct 30 19:34:16.197862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.197810    4925 scope.go:122] "RemoveContainer" containerID="ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9"
Oct 30 19:34:16.205192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.205164    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-token" pod "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" (UID: "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:16.210009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.209987    4925 scope.go:122] "RemoveContainer" containerID="ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9"
Oct 30 19:34:16.210353 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:16.210329    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9\": container with ID starting with ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9 not found: ID does not exist" containerID="ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9"
Oct 30 19:34:16.210452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.210363    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9"} err="failed to get container status \"ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9\": rpc error: code = NotFound desc = could not find container \"ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9\": container with ID starting with ef04ebad87793418240c48c617d9974b790b13d4dcc8ac23d84a225ca17adba9 not found: ID does not exist"
Oct 30 19:34:16.217102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.217061    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-podinfo" pod "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" (UID: "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:16.217102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.217070    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-kube-api-access-cwksh" pod "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" (UID: "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd"). InnerVolumeSpecName "kube-api-access-cwksh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:16.217102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.217076    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-envoy" pod "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" (UID: "4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:16.232978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.232911    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"]
Oct 30 19:34:16.239613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.239590    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-pxkg2"]
Oct 30 19:34:16.297482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297450    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.297628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297553    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.297628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297621    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c324c78-be8e-430e-adc9-3827a16a8dba-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.297777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297731    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-maas-api-tls\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.297826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297810    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.297877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297853    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-certs\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.297935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297876    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.297935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297918    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-data\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.297935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297917    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715601fa-f650-42e8-a545-ab33d4f3179c-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.298100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297950    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-metrics-certs\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.298100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.297978    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715601fa-f650-42e8-a545-ab33d4f3179c-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.298200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298108    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmvc\" (UniqueName: \"kubernetes.io/projected/7c324c78-be8e-430e-adc9-3827a16a8dba-kube-api-access-dfmvc\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.298200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298143    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.298316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298208    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.298316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298213    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kl8g\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-kube-api-access-9kl8g\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.298316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298293    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.298516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298484    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.298516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298504    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-credential-socket\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.298618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298518    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.298618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298541    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwksh\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-kube-api-access-cwksh\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.298618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298555    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.298618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298560    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715601fa-f650-42e8-a545-ab33d4f3179c-istiod-ca-cert\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.298618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.298570    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:16.300230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.300208    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-envoy\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.300580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.300562    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715601fa-f650-42e8-a545-ab33d4f3179c-istio-podinfo\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.310353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.310333    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-istio-token\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.310621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.310603    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kl8g\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-kube-api-access-9kl8g\") pod \"e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") " pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.371862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.371834    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:16.399975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.399931    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmvc\" (UniqueName: \"kubernetes.io/projected/7c324c78-be8e-430e-adc9-3827a16a8dba-kube-api-access-dfmvc\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.400197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.400180    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c324c78-be8e-430e-adc9-3827a16a8dba-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.400292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.400247    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-maas-api-tls\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.400354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.400301    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-metrics-certs\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.400968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.400926    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c324c78-be8e-430e-adc9-3827a16a8dba-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.403088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.403034    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-metrics-certs\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.403088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.403049    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-maas-api-tls\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.412635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.412605    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmvc\" (UniqueName: \"kubernetes.io/projected/7c324c78-be8e-430e-adc9-3827a16a8dba-kube-api-access-dfmvc\") pod \"maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.490427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.490373    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"]
Oct 30 19:34:16.494924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.494897    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-891d80-openshift-default-7b9544846c-z4592"]
Oct 30 19:34:16.495165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.495146    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:16.736061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.734788    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"]
Oct 30 19:34:16.737534 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:34:16.736191    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715601fa_f650_42e8_a545_ab33d4f3179c.slice/crio-e445927d93bbc29baf15f556775a11e5c1c9e9bde83dac1e04b000e1e208f9b0 WatchSource:0}: Error finding container e445927d93bbc29baf15f556775a11e5c1c9e9bde83dac1e04b000e1e208f9b0: Status 404 returned error can't find the container with id e445927d93bbc29baf15f556775a11e5c1c9e9bde83dac1e04b000e1e208f9b0
Oct 30 19:34:16.740569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.740524    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:16.740664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.740616    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:16.740664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.740657    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:16.741057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.741036    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"]
Oct 30 19:34:16.741639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.741251    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d" podUID="c3b500b6-9c01-4197-899a-3e72217112a4" containerName="maas-api" containerID="cri-o://3a048064d8349800a9823cd2e3f72fde5a9bcc0f80643a5b55b4c9ec1f29d242" gracePeriod=30
Oct 30 19:34:16.824700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.824655    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"]
Oct 30 19:34:16.834452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:16.834394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:17.030390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.030255    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:17.507748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.507714    4925 generic.go:356] "Generic (PLEG): container finished" podID="c3b500b6-9c01-4197-899a-3e72217112a4" containerID="3a048064d8349800a9823cd2e3f72fde5a9bcc0f80643a5b55b4c9ec1f29d242" exitCode=0
Oct 30 19:34:17.512053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.488349    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:17.564837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.562271    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd" path="/var/lib/kubelet/pods/4be73c9c-3cd1-40a0-8ad3-66c731ae7ecd/volumes"
Oct 30 19:34:17.564837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.563746    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0" path="/var/lib/kubelet/pods/91e2058b-e1ac-4eb8-a903-e0a1b1ee45e0/volumes"
Oct 30 19:34:17.564837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.564371    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm" event={"ID":"7c324c78-be8e-430e-adc9-3827a16a8dba","Type":"ContainerStarted","Data":"3619aba341547aec1bb8b5a5449332ab86e0bc1a4d62453b8a97b5a496c09aad"}
Oct 30 19:34:17.564837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.564448    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" event={"ID":"715601fa-f650-42e8-a545-ab33d4f3179c","Type":"ContainerStarted","Data":"581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383"}
Oct 30 19:34:17.564837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.564493    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" event={"ID":"715601fa-f650-42e8-a545-ab33d4f3179c","Type":"ContainerStarted","Data":"e445927d93bbc29baf15f556775a11e5c1c9e9bde83dac1e04b000e1e208f9b0"}
Oct 30 19:34:17.564837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.564517    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d" event={"ID":"c3b500b6-9c01-4197-899a-3e72217112a4","Type":"ContainerDied","Data":"3a048064d8349800a9823cd2e3f72fde5a9bcc0f80643a5b55b4c9ec1f29d242"}
Oct 30 19:34:17.589387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.587756    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" podStartSLOduration=37.587735086 podStartE2EDuration="37.587735086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:17.551779953 +0000 UTC m=+1692.454792991" watchObservedRunningTime="2026-10-30 19:34:17.587735086 +0000 UTC m=+1692.490748119"
Oct 30 19:34:17.622750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.617500    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:34:17.717617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.717586    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-metrics-certs\") pod \"c3b500b6-9c01-4197-899a-3e72217112a4\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") "
Oct 30 19:34:17.717940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.717908    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b500b6-9c01-4197-899a-3e72217112a4-kube-api-access-djn9h\" (UniqueName: \"kubernetes.io/projected/c3b500b6-9c01-4197-899a-3e72217112a4-kube-api-access-djn9h\") pod \"c3b500b6-9c01-4197-899a-3e72217112a4\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") "
Oct 30 19:34:17.718062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.718023    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-maas-api-tls\") pod \"c3b500b6-9c01-4197-899a-3e72217112a4\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") "
Oct 30 19:34:17.718130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.718073    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b500b6-9c01-4197-899a-3e72217112a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b500b6-9c01-4197-899a-3e72217112a4-openshift-service-ca\") pod \"c3b500b6-9c01-4197-899a-3e72217112a4\" (UID: \"c3b500b6-9c01-4197-899a-3e72217112a4\") "
Oct 30 19:34:17.718521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.718493    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b500b6-9c01-4197-899a-3e72217112a4-openshift-service-ca" pod "c3b500b6-9c01-4197-899a-3e72217112a4" (UID: "c3b500b6-9c01-4197-899a-3e72217112a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:17.719270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.719243    4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b500b6-9c01-4197-899a-3e72217112a4-openshift-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:17.720090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.720066    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-metrics-certs" pod "c3b500b6-9c01-4197-899a-3e72217112a4" (UID: "c3b500b6-9c01-4197-899a-3e72217112a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:17.720431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.720403    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-maas-api-tls" pod "c3b500b6-9c01-4197-899a-3e72217112a4" (UID: "c3b500b6-9c01-4197-899a-3e72217112a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:17.721069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.721041    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b500b6-9c01-4197-899a-3e72217112a4-kube-api-access-djn9h" pod "c3b500b6-9c01-4197-899a-3e72217112a4" (UID: "c3b500b6-9c01-4197-899a-3e72217112a4"). InnerVolumeSpecName "kube-api-access-djn9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:17.735463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.735406    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:17.821548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.821463    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djn9h\" (UniqueName: \"kubernetes.io/projected/c3b500b6-9c01-4197-899a-3e72217112a4-kube-api-access-djn9h\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:17.821548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.821494    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-maas-api-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:17.821548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.821505    4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b500b6-9c01-4197-899a-3e72217112a4-metrics-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:17.899081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:17.899043    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:18.187003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.186967    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp" podUID="68f400a1-258e-4e0e-99ea-a440a7c12939" containerName="main" containerID="cri-o://44422bd096a2a1d3c578fb3eb3357aec75724e43e36bef65e5b50dc5e0a1ee4c" gracePeriod=45
Oct 30 19:34:18.361439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.361308    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:18.377841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.377751    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:18.378180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.378151    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:18.378321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.378197    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:18.542136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.538505    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"]
Oct 30 19:34:18.542136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.539667    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c3b500b6-9c01-4197-899a-3e72217112a4" containerName="maas-api"
Oct 30 19:34:18.542136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.539705    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b500b6-9c01-4197-899a-3e72217112a4" containerName="maas-api"
Oct 30 19:34:18.542136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.539952    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b500b6-9c01-4197-899a-3e72217112a4" containerName="maas-api"
Oct 30 19:34:18.545149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.544436    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.549681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.549467    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"]
Oct 30 19:34:18.551488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.551263    4925 generic.go:356] "Generic (PLEG): container finished" podID="68f400a1-258e-4e0e-99ea-a440a7c12939" containerID="44422bd096a2a1d3c578fb3eb3357aec75724e43e36bef65e5b50dc5e0a1ee4c" exitCode=0
Oct 30 19:34:18.551488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.551454    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp" event={"ID":"68f400a1-258e-4e0e-99ea-a440a7c12939","Type":"ContainerDied","Data":"44422bd096a2a1d3c578fb3eb3357aec75724e43e36bef65e5b50dc5e0a1ee4c"}
Oct 30 19:34:18.558667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.555017    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d" event={"ID":"c3b500b6-9c01-4197-899a-3e72217112a4","Type":"ContainerDied","Data":"e4eb4bc23940046d4442530330beb79ed27f8d44c6a8a7f1a3364ad460ae444e"}
Oct 30 19:34:18.558667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.555072    4925 scope.go:122] "RemoveContainer" containerID="3a048064d8349800a9823cd2e3f72fde5a9bcc0f80643a5b55b4c9ec1f29d242"
Oct 30 19:34:18.558667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.555196    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"
Oct 30 19:34:18.575112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.560295    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm" event={"ID":"7c324c78-be8e-430e-adc9-3827a16a8dba","Type":"ContainerStarted","Data":"fabb01223cf37092c16f87c1dcb8dfc582bb8bddf688ba5c1591b814dd42f9aa"}
Oct 30 19:34:18.575112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.560662    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:18.575112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.570921    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:18.575112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.570974    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:18.645921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.645882    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40dab794-vvf58\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.646571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.646550    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrj7\" (UniqueName: \"kubernetes.io/projected/bd52578e-8671-4cda-b6f1-c08931f772a6-kube-api-access-6qrj7\") pod \"maas-api-revoke-sub-40dab794-vvf58\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.646875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.646859    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40dab794-vvf58\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.659224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.659175    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm" podStartSLOduration=77.007738314 podStartE2EDuration="1m17.659160483s" totalImagesPullingTime="651.422169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:18.650197676 +0000 UTC m=+1693.553210713" watchObservedRunningTime="2026-10-30 19:34:18.659160483 +0000 UTC m=+1693.562173520"
Oct 30 19:34:18.670045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.670020    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"]
Oct 30 19:34:18.674139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.674070    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-33b4d7-6584b6d9b4-z6z8d"]
Oct 30 19:34:18.731546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.731457    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:34:18.748025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.747999    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrj7\" (UniqueName: \"kubernetes.io/projected/bd52578e-8671-4cda-b6f1-c08931f772a6-kube-api-access-6qrj7\") pod \"maas-api-revoke-sub-40dab794-vvf58\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.748193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.748178    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40dab794-vvf58\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.748238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.748215    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40dab794-vvf58\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.748883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.748859    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40dab794-vvf58\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.750795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.750776    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40dab794-vvf58\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.773981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.763970    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrj7\" (UniqueName: \"kubernetes.io/projected/bd52578e-8671-4cda-b6f1-c08931f772a6-kube-api-access-6qrj7\") pod \"maas-api-revoke-sub-40dab794-vvf58\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.855806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.854788    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f400a1-258e-4e0e-99ea-a440a7c12939-tls-certs\" (UniqueName: \"kubernetes.io/secret/68f400a1-258e-4e0e-99ea-a440a7c12939-tls-certs\") pod \"68f400a1-258e-4e0e-99ea-a440a7c12939\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") "
Oct 30 19:34:18.855806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.854924    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-dshm\") pod \"68f400a1-258e-4e0e-99ea-a440a7c12939\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") "
Oct 30 19:34:18.855806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.854948    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-home\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-home\") pod \"68f400a1-258e-4e0e-99ea-a440a7c12939\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") "
Oct 30 19:34:18.855806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.855052    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-tmp-dir\") pod \"68f400a1-258e-4e0e-99ea-a440a7c12939\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") "
Oct 30 19:34:18.855806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.855120    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f400a1-258e-4e0e-99ea-a440a7c12939-kube-api-access-jtpbc\" (UniqueName: \"kubernetes.io/projected/68f400a1-258e-4e0e-99ea-a440a7c12939-kube-api-access-jtpbc\") pod \"68f400a1-258e-4e0e-99ea-a440a7c12939\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") "
Oct 30 19:34:18.855806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.855169    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-model-cache\") pod \"68f400a1-258e-4e0e-99ea-a440a7c12939\" (UID: \"68f400a1-258e-4e0e-99ea-a440a7c12939\") "
Oct 30 19:34:18.857520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.857196    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-model-cache" pod "68f400a1-258e-4e0e-99ea-a440a7c12939" (UID: "68f400a1-258e-4e0e-99ea-a440a7c12939"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:18.858274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.858233    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-home" pod "68f400a1-258e-4e0e-99ea-a440a7c12939" (UID: "68f400a1-258e-4e0e-99ea-a440a7c12939"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:18.858410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.858324    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-tmp-dir" pod "68f400a1-258e-4e0e-99ea-a440a7c12939" (UID: "68f400a1-258e-4e0e-99ea-a440a7c12939"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:18.861188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.861167    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-dshm" pod "68f400a1-258e-4e0e-99ea-a440a7c12939" (UID: "68f400a1-258e-4e0e-99ea-a440a7c12939"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:18.861507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.861486    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f400a1-258e-4e0e-99ea-a440a7c12939-kube-api-access-jtpbc" pod "68f400a1-258e-4e0e-99ea-a440a7c12939" (UID: "68f400a1-258e-4e0e-99ea-a440a7c12939"). InnerVolumeSpecName "kube-api-access-jtpbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:18.864449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.864413    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f400a1-258e-4e0e-99ea-a440a7c12939-tls-certs" pod "68f400a1-258e-4e0e-99ea-a440a7c12939" (UID: "68f400a1-258e-4e0e-99ea-a440a7c12939"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:18.881755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.878135    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:18.959207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.959179    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:18.959207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.959206    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:18.959386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.959222    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:18.959386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.959231    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtpbc\" (UniqueName: \"kubernetes.io/projected/68f400a1-258e-4e0e-99ea-a440a7c12939-kube-api-access-jtpbc\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:18.959386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.959242    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68f400a1-258e-4e0e-99ea-a440a7c12939-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:18.959386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:18.959254    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68f400a1-258e-4e0e-99ea-a440a7c12939-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:19.219723 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:34:19.199147    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd52578e_8671_4cda_b6f1_c08931f772a6.slice/crio-3c6d656eb2ae11a5c26b680aaa4a962672c056a67d53af80d3b2655f4496c982 WatchSource:0}: Error finding container 3c6d656eb2ae11a5c26b680aaa4a962672c056a67d53af80d3b2655f4496c982: Status 404 returned error can't find the container with id 3c6d656eb2ae11a5c26b680aaa4a962672c056a67d53af80d3b2655f4496c982
Oct 30 19:34:19.219723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.218332    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"]
Oct 30 19:34:19.265741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.265667    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b500b6-9c01-4197-899a-3e72217112a4" path="/var/lib/kubelet/pods/c3b500b6-9c01-4197-899a-3e72217112a4/volumes"
Oct 30 19:34:19.381565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.378882    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:19.381565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.378939    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:19.381565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.380499    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:19.560898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.560806    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp" podUID="68f400a1-258e-4e0e-99ea-a440a7c12939" containerName="main" probeResult="failure" output="Get \"https://10.132.0.150:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:19.571193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.571159    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58" event={"ID":"bd52578e-8671-4cda-b6f1-c08931f772a6","Type":"ContainerStarted","Data":"3c6d656eb2ae11a5c26b680aaa4a962672c056a67d53af80d3b2655f4496c982"}
Oct 30 19:34:19.572661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.572642    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"
Oct 30 19:34:19.572806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.572680    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp" event={"ID":"68f400a1-258e-4e0e-99ea-a440a7c12939","Type":"ContainerDied","Data":"4c58e8af833336e76806afe9689ef0113b29d103b7ef509160b3763c36c6716f"}
Oct 30 19:34:19.572806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.572737    4925 scope.go:122] "RemoveContainer" containerID="44422bd096a2a1d3c578fb3eb3357aec75724e43e36bef65e5b50dc5e0a1ee4c"
Oct 30 19:34:19.590293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.590271    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"]
Oct 30 19:34:19.622142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:19.622114    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-afdf16/test-model-afdf16-kserve-6db5d8b867-vjcbp"]
Oct 30 19:34:20.054753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:20.054119    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:20.376806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:20.375781    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:20.376806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:20.375834    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:20.584669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:20.584624    4925 generic.go:356] "Generic (PLEG): container finished" podID="bd52578e-8671-4cda-b6f1-c08931f772a6" containerID="64010f5a9fa183c83333409dd587136f7737d21b415e1d81a86faf72a54f3d05" exitCode=0
Oct 30 19:34:20.584831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:20.584673    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58" event={"ID":"bd52578e-8671-4cda-b6f1-c08931f772a6","Type":"ContainerDied","Data":"64010f5a9fa183c83333409dd587136f7737d21b415e1d81a86faf72a54f3d05"}
Oct 30 19:34:20.665263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:20.665227    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:21.274518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.274479    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f400a1-258e-4e0e-99ea-a440a7c12939" path="/var/lib/kubelet/pods/68f400a1-258e-4e0e-99ea-a440a7c12939/volumes"
Oct 30 19:34:21.277515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.277465    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:21.372439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.372410    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:21.372604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.372457    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:21.444799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.444759    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:21.853515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.851664    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"]
Oct 30 19:34:21.853515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.852830    4925 cpu_manager.go:414] "containerMap: removing container" podUID="68f400a1-258e-4e0e-99ea-a440a7c12939" containerName="main"
Oct 30 19:34:21.853515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.852852    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f400a1-258e-4e0e-99ea-a440a7c12939" containerName="main"
Oct 30 19:34:21.853515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.853073    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f400a1-258e-4e0e-99ea-a440a7c12939" containerName="main"
Oct 30 19:34:21.872715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.872163    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"]
Oct 30 19:34:21.872715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.872286    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:21.940803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.928595    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:21.940803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.928798    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-certs\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:21.940803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.928990    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-credential-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:21.940803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.929040    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-token\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:21.940803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.929115    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-podinfo\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:21.940803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.929165    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-data\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:21.940803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.929212    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-envoy\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:21.940803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.929239    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrmx\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-kube-api-access-wdrmx\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:21.940803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:21.929400    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212dbf9-58f5-4408-a202-d25f052ded6b-istiod-ca-cert\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.031712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.031072    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212dbf9-58f5-4408-a202-d25f052ded6b-istiod-ca-cert\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.031712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.031135    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.031712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.031220    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-certs\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.031712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.031245    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-credential-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.031712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.031273    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-token\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.031712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.031313    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-podinfo\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.031712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.031342    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-data\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.031712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.031369    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-envoy\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.031712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.031385    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrmx\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-kube-api-access-wdrmx\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.032380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.032188    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212dbf9-58f5-4408-a202-d25f052ded6b-istiod-ca-cert\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.032380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.032369    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.032711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.032514    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-certs\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.032711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.032670    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-credential-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.033341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.033313    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-data\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.038128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.036154    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-podinfo\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.038128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.036365    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-envoy\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.043883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.043855    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrmx\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-kube-api-access-wdrmx\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.046087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.046061    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-token\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.132137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.132111    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:22.163928 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:22.163885    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:34:22.248773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.248348    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-xks-ca\") pod \"bd52578e-8671-4cda-b6f1-c08931f772a6\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") "
Oct 30 19:34:22.256186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.248397    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd52578e-8671-4cda-b6f1-c08931f772a6-kube-api-access-6qrj7\" (UniqueName: \"kubernetes.io/projected/bd52578e-8671-4cda-b6f1-c08931f772a6-kube-api-access-6qrj7\") pod \"bd52578e-8671-4cda-b6f1-c08931f772a6\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") "
Oct 30 19:34:22.256186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.250279    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-openshift-ca\") pod \"bd52578e-8671-4cda-b6f1-c08931f772a6\" (UID: \"bd52578e-8671-4cda-b6f1-c08931f772a6\") "
Oct 30 19:34:22.260500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.260467    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:22.278700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.274281    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd52578e-8671-4cda-b6f1-c08931f772a6-kube-api-access-6qrj7" pod "bd52578e-8671-4cda-b6f1-c08931f772a6" (UID: "bd52578e-8671-4cda-b6f1-c08931f772a6"). InnerVolumeSpecName "kube-api-access-6qrj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:22.278700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.274364    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:22.278700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.278169    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-openshift-ca" pod "bd52578e-8671-4cda-b6f1-c08931f772a6" (UID: "bd52578e-8671-4cda-b6f1-c08931f772a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:22.283767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.283724    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:22.283767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.283756    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qrj7\" (UniqueName: \"kubernetes.io/projected/bd52578e-8671-4cda-b6f1-c08931f772a6-kube-api-access-6qrj7\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:22.284866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.284785    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-xks-ca" pod "bd52578e-8671-4cda-b6f1-c08931f772a6" (UID: "bd52578e-8671-4cda-b6f1-c08931f772a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:22.372732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.372662    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:22.372896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.372768    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:22.387620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.386617    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52578e-8671-4cda-b6f1-c08931f772a6-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:22.628073 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:34:22.628044    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4212dbf9_58f5_4408_a202_d25f052ded6b.slice/crio-b1097f668064d1a47af6bdb59bde2208014ebe42eafe7926cbeff27367532657 WatchSource:0}: Error finding container b1097f668064d1a47af6bdb59bde2208014ebe42eafe7926cbeff27367532657: Status 404 returned error can't find the container with id b1097f668064d1a47af6bdb59bde2208014ebe42eafe7926cbeff27367532657
Oct 30 19:34:22.628412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.628199    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"]
Oct 30 19:34:22.631311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.631275    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:22.631409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.631343    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:22.631409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.631374    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:22.672357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.670403    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" event={"ID":"4212dbf9-58f5-4408-a202-d25f052ded6b","Type":"ContainerStarted","Data":"b1097f668064d1a47af6bdb59bde2208014ebe42eafe7926cbeff27367532657"}
Oct 30 19:34:22.678126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.678100    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58" event={"ID":"bd52578e-8671-4cda-b6f1-c08931f772a6","Type":"ContainerDied","Data":"3c6d656eb2ae11a5c26b680aaa4a962672c056a67d53af80d3b2655f4496c982"}
Oct 30 19:34:22.678509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.678494    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c6d656eb2ae11a5c26b680aaa4a962672c056a67d53af80d3b2655f4496c982"
Oct 30 19:34:22.678627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.678368    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"
Oct 30 19:34:22.784070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:22.784037    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:23.060140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:23.060047    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:23.470709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:23.467805    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:23.470709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:23.467956    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:23.470709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:23.468163    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:23.637018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:23.636985    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:23.689377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:23.689346    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" event={"ID":"4212dbf9-58f5-4408-a202-d25f052ded6b","Type":"ContainerStarted","Data":"e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5"}
Oct 30 19:34:23.729941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:23.729840    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" podStartSLOduration=2.729826113 podStartE2EDuration="2.729826113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:23.725236207 +0000 UTC m=+1698.628249244" watchObservedRunningTime="2026-10-30 19:34:23.729826113 +0000 UTC m=+1698.632839149"
Oct 30 19:34:23.918839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:23.918795    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:24.229539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:24.229504    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:24.267069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:24.267037    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:24.268956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:24.268922    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:34:24.269092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:24.269011    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:34:24.373325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:24.373290    4925 patch_prober.go:29] interesting pod/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:24.373496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:24.373356    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:24.580617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:24.580545    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:34:25.169153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:25.168598    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:25.267117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:25.267086    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:34:25.267274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:25.267138    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:34:25.378397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:25.378364    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:25.378930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:25.378912    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:25.379639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:25.379618    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:34:26.085722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:26.083368    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:26.266896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:26.266867    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:34:26.267283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:26.266926    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:34:26.484526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:26.484485    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:26.670112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:26.670081    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:26.774228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:26.774125    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"]
Oct 30 19:34:26.774391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:26.774366    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy" containerID="cri-o://7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064" gracePeriod=30
Oct 30 19:34:26.886854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:26.886819    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:27.268665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:27.268627    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:34:27.269197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:27.268676    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:34:27.348074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:27.347522    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:28.267584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:28.267551    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:34:28.267766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:28.267611    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:34:29.264647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:29.264612    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:29.281883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:29.281853    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:34:29.282144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:29.282116    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:34:29.543449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:29.543173    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"]
Oct 30 19:34:29.633124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:29.633093    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:29.791714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:29.787493    4925 patch_prober.go:29] interesting pod/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:34:29.791714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:29.787549    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:34:29.989738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:29.987256    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"]
Oct 30 19:34:30.025629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.025589    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"]
Oct 30 19:34:30.027120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.027095    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bd52578e-8671-4cda-b6f1-c08931f772a6" containerName="revoke-keys"
Oct 30 19:34:30.027262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.027247    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd52578e-8671-4cda-b6f1-c08931f772a6" containerName="revoke-keys"
Oct 30 19:34:30.027606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.027587    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd52578e-8671-4cda-b6f1-c08931f772a6" containerName="revoke-keys"
Oct 30 19:34:30.035582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.034762    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.154240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.118560    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:30.154240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.125731    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:30.154240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.141562    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227c58e7-l9gbm\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.242441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.242334    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"]
Oct 30 19:34:30.276955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.276923    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:34:30.355750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.320296    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227c58e7-l9gbm\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.355750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.342644    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:34:30.355750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.333866    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:30.355750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.342908    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227c58e7-l9gbm\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.355750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.342965    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxhxd\" (UniqueName: \"kubernetes.io/projected/bf4959ec-c75d-42e9-95ca-433649f375bb-kube-api-access-nxhxd\") pod \"maas-api-revoke-sub-227c58e7-l9gbm\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.355750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.321036    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227c58e7-l9gbm\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.445284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.445254    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227c58e7-l9gbm\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.445468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.445301    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxhxd\" (UniqueName: \"kubernetes.io/projected/bf4959ec-c75d-42e9-95ca-433649f375bb-kube-api-access-nxhxd\") pod \"maas-api-revoke-sub-227c58e7-l9gbm\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.448293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.448262    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227c58e7-l9gbm\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.457696 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.457662    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxhxd\" (UniqueName: \"kubernetes.io/projected/bf4959ec-c75d-42e9-95ca-433649f375bb-kube-api-access-nxhxd\") pod \"maas-api-revoke-sub-227c58e7-l9gbm\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.542130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.542037    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:30.741746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.730601    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:30.840312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:30.840279    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"]
Oct 30 19:34:30.841884 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:34:30.841852    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4959ec_c75d_42e9_95ca_433649f375bb.slice/crio-32f7c74ebd824e65a7bb0dce95e4f1bfd560b6536c77b7f9515501e21329d773 WatchSource:0}: Error finding container 32f7c74ebd824e65a7bb0dce95e4f1bfd560b6536c77b7f9515501e21329d773: Status 404 returned error can't find the container with id 32f7c74ebd824e65a7bb0dce95e4f1bfd560b6536c77b7f9515501e21329d773
Oct 30 19:34:31.272050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:31.272011    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:31.272521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:31.272493    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:31.273192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:31.273173    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:34:31.725729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:31.723988    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:31.840369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:31.840341    4925 generic.go:356] "Generic (PLEG): container finished" podID="bf4959ec-c75d-42e9-95ca-433649f375bb" containerID="d408320da678d81d86d3777131195704d417791ea8dffecf1b26b265abb9e56b" exitCode=0
Oct 30 19:34:31.840516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:31.840424    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm" event={"ID":"bf4959ec-c75d-42e9-95ca-433649f375bb","Type":"ContainerDied","Data":"d408320da678d81d86d3777131195704d417791ea8dffecf1b26b265abb9e56b"}
Oct 30 19:34:31.840516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:31.840477    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm" event={"ID":"bf4959ec-c75d-42e9-95ca-433649f375bb","Type":"ContainerStarted","Data":"32f7c74ebd824e65a7bb0dce95e4f1bfd560b6536c77b7f9515501e21329d773"}
Oct 30 19:34:32.072919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.072895    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:34:32.174024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.173994    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-envoy\") pod \"febce7a9-ce87-4830-9d20-9ace649530e2\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") "
Oct 30 19:34:32.174209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174106    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-socket\") pod \"febce7a9-ce87-4830-9d20-9ace649530e2\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") "
Oct 30 19:34:32.174209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174134    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-data\") pod \"febce7a9-ce87-4830-9d20-9ace649530e2\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") "
Oct 30 19:34:32.174366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174304    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-kube-api-access-bhf2p\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-kube-api-access-bhf2p\") pod \"febce7a9-ce87-4830-9d20-9ace649530e2\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") "
Oct 30 19:34:32.174405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174378    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/febce7a9-ce87-4830-9d20-9ace649530e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febce7a9-ce87-4830-9d20-9ace649530e2-istio-podinfo\") pod \"febce7a9-ce87-4830-9d20-9ace649530e2\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") "
Oct 30 19:34:32.174446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174426    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-credential-socket\") pod \"febce7a9-ce87-4830-9d20-9ace649530e2\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") "
Oct 30 19:34:32.174507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174439    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-socket" pod "febce7a9-ce87-4830-9d20-9ace649530e2" (UID: "febce7a9-ce87-4830-9d20-9ace649530e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.174507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174483    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/febce7a9-ce87-4830-9d20-9ace649530e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febce7a9-ce87-4830-9d20-9ace649530e2-istiod-ca-cert\") pod \"febce7a9-ce87-4830-9d20-9ace649530e2\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") "
Oct 30 19:34:32.174588 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174485    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-data" pod "febce7a9-ce87-4830-9d20-9ace649530e2" (UID: "febce7a9-ce87-4830-9d20-9ace649530e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.174700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174667    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-credential-socket" pod "febce7a9-ce87-4830-9d20-9ace649530e2" (UID: "febce7a9-ce87-4830-9d20-9ace649530e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.174783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174679    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-istio-token\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-istio-token\") pod \"febce7a9-ce87-4830-9d20-9ace649530e2\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") "
Oct 30 19:34:32.174847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174817    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-certs\") pod \"febce7a9-ce87-4830-9d20-9ace649530e2\" (UID: \"febce7a9-ce87-4830-9d20-9ace649530e2\") "
Oct 30 19:34:32.174908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.174855    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/febce7a9-ce87-4830-9d20-9ace649530e2-istiod-ca-cert" pod "febce7a9-ce87-4830-9d20-9ace649530e2" (UID: "febce7a9-ce87-4830-9d20-9ace649530e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:32.175454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.175350    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-certs" pod "febce7a9-ce87-4830-9d20-9ace649530e2" (UID: "febce7a9-ce87-4830-9d20-9ace649530e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.176042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.176017    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.176187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.176045    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.176187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.176059    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.176187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.176072    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.176187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.176089    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febce7a9-ce87-4830-9d20-9ace649530e2-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.176586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.176563    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-envoy" pod "febce7a9-ce87-4830-9d20-9ace649530e2" (UID: "febce7a9-ce87-4830-9d20-9ace649530e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:32.177036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.177017    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-kube-api-access-bhf2p" pod "febce7a9-ce87-4830-9d20-9ace649530e2" (UID: "febce7a9-ce87-4830-9d20-9ace649530e2"). InnerVolumeSpecName "kube-api-access-bhf2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:32.177109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.177071    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-istio-token" pod "febce7a9-ce87-4830-9d20-9ace649530e2" (UID: "febce7a9-ce87-4830-9d20-9ace649530e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:32.177148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.177121    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/febce7a9-ce87-4830-9d20-9ace649530e2-istio-podinfo" pod "febce7a9-ce87-4830-9d20-9ace649530e2" (UID: "febce7a9-ce87-4830-9d20-9ace649530e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:32.278105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.278027    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febce7a9-ce87-4830-9d20-9ace649530e2-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.278105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.278053    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.278105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.278064    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febce7a9-ce87-4830-9d20-9ace649530e2-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.278105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.278075    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhf2p\" (UniqueName: \"kubernetes.io/projected/febce7a9-ce87-4830-9d20-9ace649530e2-kube-api-access-bhf2p\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.285590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.285553    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:32.456036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.456006    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:32.619565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.619481    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"]
Oct 30 19:34:32.619868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.619842    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy" containerID="cri-o://18bc57ad37fe73844e365a6529054189c023f9b0f24d6c263c0ca85efe1c3443" gracePeriod=30
Oct 30 19:34:32.654797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.654757    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:32.836714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.835823    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:33.000778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.993446    4925 generic.go:356] "Generic (PLEG): container finished" podID="febce7a9-ce87-4830-9d20-9ace649530e2" containerID="7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064" exitCode=0
Oct 30 19:34:33.000778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.993556    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"
Oct 30 19:34:33.000778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.995095    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" event={"ID":"febce7a9-ce87-4830-9d20-9ace649530e2","Type":"ContainerDied","Data":"7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064"}
Oct 30 19:34:33.000778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.995182    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt" event={"ID":"febce7a9-ce87-4830-9d20-9ace649530e2","Type":"ContainerDied","Data":"835f69b807acced267c68cf1a0286133e5eacd822a195e4cc2cbff5c8c50ed24"}
Oct 30 19:34:33.000778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:32.995225    4925 scope.go:122] "RemoveContainer" containerID="7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064"
Oct 30 19:34:33.086478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.086049    4925 scope.go:122] "RemoveContainer" containerID="6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b"
Oct 30 19:34:33.100157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.100134    4925 scope.go:122] "RemoveContainer" containerID="7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064"
Oct 30 19:34:33.100448 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:33.100428    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064\": container with ID starting with 7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064 not found: ID does not exist" containerID="7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064"
Oct 30 19:34:33.100527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.100457    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064"} err="failed to get container status \"7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064\": rpc error: code = NotFound desc = could not find container \"7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064\": container with ID starting with 7bea2251ba87f6c0e396bdc1d998a4dfb45fd59732d32a87c340b8b572695064 not found: ID does not exist"
Oct 30 19:34:33.100527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.100474    4925 scope.go:122] "RemoveContainer" containerID="6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b"
Oct 30 19:34:33.100849 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:33.100831    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b\": container with ID starting with 6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b not found: ID does not exist" containerID="6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b"
Oct 30 19:34:33.100940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.100850    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b"} err="failed to get container status \"6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b\": rpc error: code = NotFound desc = could not find container \"6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b\": container with ID starting with 6628e9ef382f709eaecce3a0374f5d62dac67ee4ac30c8c76f1a208babbbf29b not found: ID does not exist"
Oct 30 19:34:33.122172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.122142    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"]
Oct 30 19:34:33.157001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.156961    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-33b4d7-openshift-default-5c9994896f-hvxlt"]
Oct 30 19:34:33.262680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.262591    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:33.269058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.269032    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" path="/var/lib/kubelet/pods/febce7a9-ce87-4830-9d20-9ace649530e2/volumes"
Oct 30 19:34:33.357927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.357885    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4959ec-c75d-42e9-95ca-433649f375bb-kube-api-access-nxhxd\" (UniqueName: \"kubernetes.io/projected/bf4959ec-c75d-42e9-95ca-433649f375bb-kube-api-access-nxhxd\") pod \"bf4959ec-c75d-42e9-95ca-433649f375bb\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") "
Oct 30 19:34:33.360319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.360258    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:33.360796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.360776    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-xks-ca\") pod \"bf4959ec-c75d-42e9-95ca-433649f375bb\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") "
Oct 30 19:34:33.362044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.362007    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-openshift-ca" pod "bf4959ec-c75d-42e9-95ca-433649f375bb" (UID: "bf4959ec-c75d-42e9-95ca-433649f375bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:33.362198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.362179    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-openshift-ca\") pod \"bf4959ec-c75d-42e9-95ca-433649f375bb\" (UID: \"bf4959ec-c75d-42e9-95ca-433649f375bb\") "
Oct 30 19:34:33.392516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.380796    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:33.438749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.426803    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-xks-ca" pod "bf4959ec-c75d-42e9-95ca-433649f375bb" (UID: "bf4959ec-c75d-42e9-95ca-433649f375bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:33.471907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.471867    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4959ec-c75d-42e9-95ca-433649f375bb-kube-api-access-nxhxd" pod "bf4959ec-c75d-42e9-95ca-433649f375bb" (UID: "bf4959ec-c75d-42e9-95ca-433649f375bb"). InnerVolumeSpecName "kube-api-access-nxhxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:33.531767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.523649    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf4959ec-c75d-42e9-95ca-433649f375bb-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:33.531767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.523707    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxhxd\" (UniqueName: \"kubernetes.io/projected/bf4959ec-c75d-42e9-95ca-433649f375bb-kube-api-access-nxhxd\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:33.637714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.636269    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:33.834774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:33.820229    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:34.008003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:34.007977    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"
Oct 30 19:34:34.008481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:34.008019    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm" event={"ID":"bf4959ec-c75d-42e9-95ca-433649f375bb","Type":"ContainerDied","Data":"32f7c74ebd824e65a7bb0dce95e4f1bfd560b6536c77b7f9515501e21329d773"}
Oct 30 19:34:34.008481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:34.008061    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32f7c74ebd824e65a7bb0dce95e4f1bfd560b6536c77b7f9515501e21329d773"
Oct 30 19:34:34.082463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:34.082425    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:34.300757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:34.286193    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:35.230463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:35.230426    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:36.174777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:36.174370    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:37.093789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:37.093757    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:37.876914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:37.872623    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:38.102761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.099673    4925 generic.go:356] "Generic (PLEG): container finished" podID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerID="18bc57ad37fe73844e365a6529054189c023f9b0f24d6c263c0ca85efe1c3443" exitCode=0
Oct 30 19:34:38.104867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.103291    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" event={"ID":"b5a0d956-67cf-40d2-a0b0-427786ce6161","Type":"ContainerDied","Data":"18bc57ad37fe73844e365a6529054189c023f9b0f24d6c263c0ca85efe1c3443"}
Oct 30 19:34:38.104867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.103369    4925 scope.go:122] "RemoveContainer" containerID="1cd424c0dc77ab3dda6417736cb8e6cc04aa157d640a119f13f08e2116839064"
Oct 30 19:34:38.194063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.194029    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:38.240946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.240917    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:34:38.352260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352177    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-envoy\") pod \"b5a0d956-67cf-40d2-a0b0-427786ce6161\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") "
Oct 30 19:34:38.352260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352253    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-data\") pod \"b5a0d956-67cf-40d2-a0b0-427786ce6161\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") "
Oct 30 19:34:38.352525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352367    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-podinfo\") pod \"b5a0d956-67cf-40d2-a0b0-427786ce6161\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") "
Oct 30 19:34:38.352525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352492    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-data" pod "b5a0d956-67cf-40d2-a0b0-427786ce6161" (UID: "b5a0d956-67cf-40d2-a0b0-427786ce6161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.352525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352499    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-socket\") pod \"b5a0d956-67cf-40d2-a0b0-427786ce6161\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") "
Oct 30 19:34:38.352667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352631    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-credential-socket\") pod \"b5a0d956-67cf-40d2-a0b0-427786ce6161\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") "
Oct 30 19:34:38.352784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352751    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a0d956-67cf-40d2-a0b0-427786ce6161-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0d956-67cf-40d2-a0b0-427786ce6161-istiod-ca-cert\") pod \"b5a0d956-67cf-40d2-a0b0-427786ce6161\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") "
Oct 30 19:34:38.352850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352774    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-socket" pod "b5a0d956-67cf-40d2-a0b0-427786ce6161" (UID: "b5a0d956-67cf-40d2-a0b0-427786ce6161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.352909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352892    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-token\") pod \"b5a0d956-67cf-40d2-a0b0-427786ce6161\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") "
Oct 30 19:34:38.352971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.352937    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-certs\") pod \"b5a0d956-67cf-40d2-a0b0-427786ce6161\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") "
Oct 30 19:34:38.353052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.353027    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-kube-api-access-z59hm\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-kube-api-access-z59hm\") pod \"b5a0d956-67cf-40d2-a0b0-427786ce6161\" (UID: \"b5a0d956-67cf-40d2-a0b0-427786ce6161\") "
Oct 30 19:34:38.353278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.353254    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-certs" pod "b5a0d956-67cf-40d2-a0b0-427786ce6161" (UID: "b5a0d956-67cf-40d2-a0b0-427786ce6161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.353357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.353300    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-credential-socket" pod "b5a0d956-67cf-40d2-a0b0-427786ce6161" (UID: "b5a0d956-67cf-40d2-a0b0-427786ce6161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.354092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.354060    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.354214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.354179    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.354214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.354198    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.354362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.354214    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.354362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.354141    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a0d956-67cf-40d2-a0b0-427786ce6161-istiod-ca-cert" pod "b5a0d956-67cf-40d2-a0b0-427786ce6161" (UID: "b5a0d956-67cf-40d2-a0b0-427786ce6161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:38.355798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.355774    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-token" pod "b5a0d956-67cf-40d2-a0b0-427786ce6161" (UID: "b5a0d956-67cf-40d2-a0b0-427786ce6161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:38.355798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.355785    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-envoy" pod "b5a0d956-67cf-40d2-a0b0-427786ce6161" (UID: "b5a0d956-67cf-40d2-a0b0-427786ce6161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:38.355976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.355805    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-kube-api-access-z59hm" pod "b5a0d956-67cf-40d2-a0b0-427786ce6161" (UID: "b5a0d956-67cf-40d2-a0b0-427786ce6161"). InnerVolumeSpecName "kube-api-access-z59hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:38.356303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.356289    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-podinfo" pod "b5a0d956-67cf-40d2-a0b0-427786ce6161" (UID: "b5a0d956-67cf-40d2-a0b0-427786ce6161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:38.455795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.455766    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.455795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.455794    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z59hm\" (UniqueName: \"kubernetes.io/projected/b5a0d956-67cf-40d2-a0b0-427786ce6161-kube-api-access-z59hm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.455969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.455805    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.455969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.455813    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a0d956-67cf-40d2-a0b0-427786ce6161-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.455969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.455822    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a0d956-67cf-40d2-a0b0-427786ce6161-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:38.581718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.575343    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:38.861641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:38.861604    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:39.060542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:39.060504    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:39.116585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:39.116514    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"
Oct 30 19:34:39.116585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:39.116570    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6" event={"ID":"b5a0d956-67cf-40d2-a0b0-427786ce6161","Type":"ContainerDied","Data":"cfa25027fd8f5420b55eee218e4b205d08467dfd34b52751ef91496d4435e35f"}
Oct 30 19:34:39.117227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:39.116630    4925 scope.go:122] "RemoveContainer" containerID="18bc57ad37fe73844e365a6529054189c023f9b0f24d6c263c0ca85efe1c3443"
Oct 30 19:34:39.150363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:39.150337    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"]
Oct 30 19:34:39.158725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:39.158699    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ef5dca-openshift-default-b7b5597b9-z8tl6"]
Oct 30 19:34:39.244898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:39.244866    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:39.266703 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:39.266664    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" path="/var/lib/kubelet/pods/b5a0d956-67cf-40d2-a0b0-427786ce6161/volumes"
Oct 30 19:34:39.482709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:39.470349    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:40.249647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:40.249615    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:40.481800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:40.459774    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:41.343656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:41.343615    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:42.077970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:42.077931    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:42.918751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:42.918712    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:43.689753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:43.689709    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:44.411712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:44.409988    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:44.610729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:44.608275    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8" podUID="f18e20c5-6289-4bd8-9706-11d6008895ee" containerName="main" containerID="cri-o://3127655b5f7cb721bba7fccb639f6f78ddd0f136cb23d8e851e14c6a470ed899" gracePeriod=45
Oct 30 19:34:45.010195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.010060    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:45.255098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.234002    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:45.579668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.579621    4925 generic.go:356] "Generic (PLEG): container finished" podID="f18e20c5-6289-4bd8-9706-11d6008895ee" containerID="3127655b5f7cb721bba7fccb639f6f78ddd0f136cb23d8e851e14c6a470ed899" exitCode=0
Oct 30 19:34:45.618051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.616098    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh" podUID="8de41fc9-e79a-4141-b3a1-10934169a976" containerName="main" containerID="cri-o://b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930" gracePeriod=45
Oct 30 19:34:45.640641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.640604    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8" event={"ID":"f18e20c5-6289-4bd8-9706-11d6008895ee","Type":"ContainerDied","Data":"3127655b5f7cb721bba7fccb639f6f78ddd0f136cb23d8e851e14c6a470ed899"}
Oct 30 19:34:45.641055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.641036    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"]
Oct 30 19:34:45.647485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.647454    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:34:45.648226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.647926    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy"
Oct 30 19:34:45.648364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.648349    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy"
Oct 30 19:34:45.648461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.648450    4925 cpu_manager.go:414] "containerMap: removing container" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy"
Oct 30 19:34:45.648538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.648528    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy"
Oct 30 19:34:45.648780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.648680    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy"
Oct 30 19:34:45.648896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.648883    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy"
Oct 30 19:34:45.649002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.648992    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bf4959ec-c75d-42e9-95ca-433649f375bb" containerName="revoke-keys"
Oct 30 19:34:45.649074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.649063    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4959ec-c75d-42e9-95ca-433649f375bb" containerName="revoke-keys"
Oct 30 19:34:45.650059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.650041    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy"
Oct 30 19:34:45.651916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.651899    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy"
Oct 30 19:34:45.652133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.652118    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18e20c5-6289-4bd8-9706-11d6008895ee" containerName="main"
Oct 30 19:34:45.652303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.652288    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy"
Oct 30 19:34:45.658860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.658847    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4959ec-c75d-42e9-95ca-433649f375bb" containerName="revoke-keys"
Oct 30 19:34:45.681712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.666913    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"]
Oct 30 19:34:45.681712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.666954    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"]
Oct 30 19:34:45.681712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.667269    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:45.681712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.680980    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:34:45.681712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.681041    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:45.761428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.761399    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-home\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-home\") pod \"f18e20c5-6289-4bd8-9706-11d6008895ee\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") "
Oct 30 19:34:45.761638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.761619    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-model-cache\") pod \"f18e20c5-6289-4bd8-9706-11d6008895ee\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") "
Oct 30 19:34:45.761735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.761653    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-home" pod "f18e20c5-6289-4bd8-9706-11d6008895ee" (UID: "f18e20c5-6289-4bd8-9706-11d6008895ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:45.761735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.761679    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-tmp-dir\") pod \"f18e20c5-6289-4bd8-9706-11d6008895ee\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") "
Oct 30 19:34:45.761935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.761914    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-tmp-dir" pod "f18e20c5-6289-4bd8-9706-11d6008895ee" (UID: "f18e20c5-6289-4bd8-9706-11d6008895ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:45.761993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.761927    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-model-cache" pod "f18e20c5-6289-4bd8-9706-11d6008895ee" (UID: "f18e20c5-6289-4bd8-9706-11d6008895ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:45.761993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.761966    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f18e20c5-6289-4bd8-9706-11d6008895ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/f18e20c5-6289-4bd8-9706-11d6008895ee-tls-certs\") pod \"f18e20c5-6289-4bd8-9706-11d6008895ee\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") "
Oct 30 19:34:45.762065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.762010    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18e20c5-6289-4bd8-9706-11d6008895ee-kube-api-access-qrsg8\" (UniqueName: \"kubernetes.io/projected/f18e20c5-6289-4bd8-9706-11d6008895ee-kube-api-access-qrsg8\") pod \"f18e20c5-6289-4bd8-9706-11d6008895ee\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") "
Oct 30 19:34:45.762065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.762056    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-dshm\") pod \"f18e20c5-6289-4bd8-9706-11d6008895ee\" (UID: \"f18e20c5-6289-4bd8-9706-11d6008895ee\") "
Oct 30 19:34:45.762539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.762509    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:45.762934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.762914    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwcmk\" (UniqueName: \"kubernetes.io/projected/f0e5665f-cc81-41e7-9c87-1be8941e999d-kube-api-access-lwcmk\") pod \"maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:45.763045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.762978    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:45.763202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.763183    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:45.763274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.763208    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:45.763274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.763245    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:45.764963 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.764942    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18e20c5-6289-4bd8-9706-11d6008895ee-kube-api-access-qrsg8" pod "f18e20c5-6289-4bd8-9706-11d6008895ee" (UID: "f18e20c5-6289-4bd8-9706-11d6008895ee"). InnerVolumeSpecName "kube-api-access-qrsg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:45.770271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.770240    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-dshm" pod "f18e20c5-6289-4bd8-9706-11d6008895ee" (UID: "f18e20c5-6289-4bd8-9706-11d6008895ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:45.773718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.773674    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:45.773867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.773811    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18e20c5-6289-4bd8-9706-11d6008895ee-tls-certs" pod "f18e20c5-6289-4bd8-9706-11d6008895ee" (UID: "f18e20c5-6289-4bd8-9706-11d6008895ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:45.880761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.869022    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwcmk\" (UniqueName: \"kubernetes.io/projected/f0e5665f-cc81-41e7-9c87-1be8941e999d-kube-api-access-lwcmk\") pod \"maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:45.880761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.869103    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:45.880761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.869316    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:45.880761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.869495    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18e20c5-6289-4bd8-9706-11d6008895ee-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:45.880761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.869516    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrsg8\" (UniqueName: \"kubernetes.io/projected/f18e20c5-6289-4bd8-9706-11d6008895ee-kube-api-access-qrsg8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:45.880761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.869530    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18e20c5-6289-4bd8-9706-11d6008895ee-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:45.880761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.870334    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:45.980864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:45.979021    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwcmk\" (UniqueName: \"kubernetes.io/projected/f0e5665f-cc81-41e7-9c87-1be8941e999d-kube-api-access-lwcmk\") pod \"maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:46.061253 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.061172    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:46.189345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.189316    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:34:46.222745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.220234    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-model-cache\") pod \"8de41fc9-e79a-4141-b3a1-10934169a976\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") "
Oct 30 19:34:46.222745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.220309    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-dshm\") pod \"8de41fc9-e79a-4141-b3a1-10934169a976\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") "
Oct 30 19:34:46.222745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.220384    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-tmp-dir\") pod \"8de41fc9-e79a-4141-b3a1-10934169a976\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") "
Oct 30 19:34:46.222745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.220516    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de41fc9-e79a-4141-b3a1-10934169a976-kube-api-access-h65lq\" (UniqueName: \"kubernetes.io/projected/8de41fc9-e79a-4141-b3a1-10934169a976-kube-api-access-h65lq\") pod \"8de41fc9-e79a-4141-b3a1-10934169a976\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") "
Oct 30 19:34:46.222745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.220622    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8de41fc9-e79a-4141-b3a1-10934169a976-tls-certs\" (UniqueName: \"kubernetes.io/secret/8de41fc9-e79a-4141-b3a1-10934169a976-tls-certs\") pod \"8de41fc9-e79a-4141-b3a1-10934169a976\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") "
Oct 30 19:34:46.222745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.220729    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-home\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-home\") pod \"8de41fc9-e79a-4141-b3a1-10934169a976\" (UID: \"8de41fc9-e79a-4141-b3a1-10934169a976\") "
Oct 30 19:34:46.222745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.222731    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-model-cache" pod "8de41fc9-e79a-4141-b3a1-10934169a976" (UID: "8de41fc9-e79a-4141-b3a1-10934169a976"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:46.223652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.223536    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-home" pod "8de41fc9-e79a-4141-b3a1-10934169a976" (UID: "8de41fc9-e79a-4141-b3a1-10934169a976"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:46.225304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.224602    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-tmp-dir" pod "8de41fc9-e79a-4141-b3a1-10934169a976" (UID: "8de41fc9-e79a-4141-b3a1-10934169a976"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:46.231494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.230080    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-dshm" pod "8de41fc9-e79a-4141-b3a1-10934169a976" (UID: "8de41fc9-e79a-4141-b3a1-10934169a976"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:46.232298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.232253    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de41fc9-e79a-4141-b3a1-10934169a976-tls-certs" pod "8de41fc9-e79a-4141-b3a1-10934169a976" (UID: "8de41fc9-e79a-4141-b3a1-10934169a976"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:46.237268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.237231    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de41fc9-e79a-4141-b3a1-10934169a976-kube-api-access-h65lq" pod "8de41fc9-e79a-4141-b3a1-10934169a976" (UID: "8de41fc9-e79a-4141-b3a1-10934169a976"). InnerVolumeSpecName "kube-api-access-h65lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:46.259262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.259225    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8" podUID="f18e20c5-6289-4bd8-9706-11d6008895ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:46.323605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.323566    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h65lq\" (UniqueName: \"kubernetes.io/projected/8de41fc9-e79a-4141-b3a1-10934169a976-kube-api-access-h65lq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:46.323605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.323601    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de41fc9-e79a-4141-b3a1-10934169a976-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:46.323605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.323614    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:46.323916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.323628    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:46.323916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.323641    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:46.323916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.323656    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de41fc9-e79a-4141-b3a1-10934169a976-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:46.338679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.338649    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:46.432938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.428778    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:46.600089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.593316    4925 generic.go:356] "Generic (PLEG): container finished" podID="8de41fc9-e79a-4141-b3a1-10934169a976" containerID="b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930" exitCode=0
Oct 30 19:34:46.600089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.593404    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"
Oct 30 19:34:46.600089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.593489    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh" event={"ID":"8de41fc9-e79a-4141-b3a1-10934169a976","Type":"ContainerDied","Data":"b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930"}
Oct 30 19:34:46.600089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.593525    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh" event={"ID":"8de41fc9-e79a-4141-b3a1-10934169a976","Type":"ContainerDied","Data":"0bcf304d0b687fc565647efec6e07b85c00874b78af2406477ca4ba7a4a430ff"}
Oct 30 19:34:46.600089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.593546    4925 scope.go:122] "RemoveContainer" containerID="b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930"
Oct 30 19:34:46.603394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.603364    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8" event={"ID":"f18e20c5-6289-4bd8-9706-11d6008895ee","Type":"ContainerDied","Data":"dd7fc670899487b088f509c86492ff0ad12450622a831a3495187d879b8f3494"}
Oct 30 19:34:46.603544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.603454    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"
Oct 30 19:34:46.613777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.613756    4925 scope.go:122] "RemoveContainer" containerID="b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930"
Oct 30 19:34:46.614106 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:46.614084    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930\": container with ID starting with b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930 not found: ID does not exist" containerID="b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930"
Oct 30 19:34:46.614207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.614114    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930"} err="failed to get container status \"b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930\": rpc error: code = NotFound desc = could not find container \"b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930\": container with ID starting with b0a988dd26e22760559842de30d159a203a67d5ea4eafd9d7899346317c03930 not found: ID does not exist"
Oct 30 19:34:46.614207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.614135    4925 scope.go:122] "RemoveContainer" containerID="3127655b5f7cb721bba7fccb639f6f78ddd0f136cb23d8e851e14c6a470ed899"
Oct 30 19:34:46.629657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.629632    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"]
Oct 30 19:34:46.633787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.633765    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d52364/test-model-d52364-kserve-59db4bcd7f-6c7fh"]
Oct 30 19:34:46.640649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.640625    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"]
Oct 30 19:34:46.646994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.646969    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-02feaa/test-model-02feaa-kserve-7944dcdf6f-llwd8"]
Oct 30 19:34:46.742165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:46.742133    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"]
Oct 30 19:34:46.742625 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:34:46.742601    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e5665f_cc81_41e7_9c87_1be8941e999d.slice/crio-e59a9ccb97e9cfb35e9328206159df7ab271734bafd511472c9245fd1fc7ebfd WatchSource:0}: Error finding container e59a9ccb97e9cfb35e9328206159df7ab271734bafd511472c9245fd1fc7ebfd: Status 404 returned error can't find the container with id e59a9ccb97e9cfb35e9328206159df7ab271734bafd511472c9245fd1fc7ebfd
Oct 30 19:34:47.215035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.215007    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"]
Oct 30 19:34:47.215788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.215773    4925 cpu_manager.go:414] "containerMap: removing container" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy"
Oct 30 19:34:47.215874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.215789    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febce7a9-ce87-4830-9d20-9ace649530e2" containerName="istio-proxy"
Oct 30 19:34:47.215874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.215804    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8de41fc9-e79a-4141-b3a1-10934169a976" containerName="main"
Oct 30 19:34:47.215874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.215809    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de41fc9-e79a-4141-b3a1-10934169a976" containerName="main"
Oct 30 19:34:47.215874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.215821    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f18e20c5-6289-4bd8-9706-11d6008895ee" containerName="main"
Oct 30 19:34:47.215874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.215826    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18e20c5-6289-4bd8-9706-11d6008895ee" containerName="main"
Oct 30 19:34:47.216034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.215980    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a0d956-67cf-40d2-a0b0-427786ce6161" containerName="istio-proxy"
Oct 30 19:34:47.216034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.215989    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de41fc9-e79a-4141-b3a1-10934169a976" containerName="main"
Oct 30 19:34:47.219767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.219748    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.227702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.227575    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"]
Oct 30 19:34:47.249090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.249065    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35a893e-rgwsk\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.249237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.249103    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35a893e-rgwsk\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.249387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.249364    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8fvm\" (UniqueName: \"kubernetes.io/projected/3a059da2-0698-4630-bbba-b85e37e713f6-kube-api-access-f8fvm\") pod \"maas-api-revoke-sub-c35a893e-rgwsk\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.269135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.269106    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de41fc9-e79a-4141-b3a1-10934169a976" path="/var/lib/kubelet/pods/8de41fc9-e79a-4141-b3a1-10934169a976/volumes"
Oct 30 19:34:47.269956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.269933    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f18e20c5-6289-4bd8-9706-11d6008895ee" path="/var/lib/kubelet/pods/f18e20c5-6289-4bd8-9706-11d6008895ee/volumes"
Oct 30 19:34:47.354938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.354894    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fvm\" (UniqueName: \"kubernetes.io/projected/3a059da2-0698-4630-bbba-b85e37e713f6-kube-api-access-f8fvm\") pod \"maas-api-revoke-sub-c35a893e-rgwsk\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.359956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.359927    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35a893e-rgwsk\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.361054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.361030    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35a893e-rgwsk\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.361704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.357622    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:47.361955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.360890    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35a893e-rgwsk\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.385894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.385807    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35a893e-rgwsk\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.388970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.388941    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fvm\" (UniqueName: \"kubernetes.io/projected/3a059da2-0698-4630-bbba-b85e37e713f6-kube-api-access-f8fvm\") pod \"maas-api-revoke-sub-c35a893e-rgwsk\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.538189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.538164    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:47.622161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.622127    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2" event={"ID":"f0e5665f-cc81-41e7-9c87-1be8941e999d","Type":"ContainerStarted","Data":"d2488e1dff079bf21cc412c36630e73ddcc75698c932c7044de9c922e22b9a2a"}
Oct 30 19:34:47.622516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.622190    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2" event={"ID":"f0e5665f-cc81-41e7-9c87-1be8941e999d","Type":"ContainerStarted","Data":"e59a9ccb97e9cfb35e9328206159df7ab271734bafd511472c9245fd1fc7ebfd"}
Oct 30 19:34:47.831085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.831030    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2" podStartSLOduration=2.8310149300000003 podStartE2EDuration="2.83101493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:47.641883242 +0000 UTC m=+1722.544896286" watchObservedRunningTime="2026-10-30 19:34:47.83101493 +0000 UTC m=+1722.734027967"
Oct 30 19:34:47.832892 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:34:47.832539    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a059da2_0698_4630_bbba_b85e37e713f6.slice/crio-5ec7d6b26bf2a3bc0508a71ee0d6e8c136cdfe57a051426a78840dafc27f7de2 WatchSource:0}: Error finding container 5ec7d6b26bf2a3bc0508a71ee0d6e8c136cdfe57a051426a78840dafc27f7de2: Status 404 returned error can't find the container with id 5ec7d6b26bf2a3bc0508a71ee0d6e8c136cdfe57a051426a78840dafc27f7de2
Oct 30 19:34:47.832892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:47.832562    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"]
Oct 30 19:34:48.027162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:48.027122    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:48.198712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:48.196833    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:48.423115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:48.423081    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:48.634706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:48.634602    4925 generic.go:356] "Generic (PLEG): container finished" podID="f0e5665f-cc81-41e7-9c87-1be8941e999d" containerID="d2488e1dff079bf21cc412c36630e73ddcc75698c932c7044de9c922e22b9a2a" exitCode=0
Oct 30 19:34:48.635105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:48.634682    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2" event={"ID":"f0e5665f-cc81-41e7-9c87-1be8941e999d","Type":"ContainerDied","Data":"d2488e1dff079bf21cc412c36630e73ddcc75698c932c7044de9c922e22b9a2a"}
Oct 30 19:34:48.636715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:48.636673    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk" event={"ID":"3a059da2-0698-4630-bbba-b85e37e713f6","Type":"ContainerStarted","Data":"0dc5ed479914afed937d00d1cf0c9e2400ee04d1ae3a769982eae70c95b592ec"}
Oct 30 19:34:48.636871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:48.636728    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk" event={"ID":"3a059da2-0698-4630-bbba-b85e37e713f6","Type":"ContainerStarted","Data":"5ec7d6b26bf2a3bc0508a71ee0d6e8c136cdfe57a051426a78840dafc27f7de2"}
Oct 30 19:34:48.670062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:48.670007    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk" podStartSLOduration=1.669988488 podStartE2EDuration="1.669988488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:48.666663175 +0000 UTC m=+1723.569676213" watchObservedRunningTime="2026-10-30 19:34:48.669988488 +0000 UTC m=+1723.573001527"
Oct 30 19:34:48.837725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:48.837304    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:49.206353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.206307    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:49.542718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.538360    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:49.647821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.647789    4925 generic.go:356] "Generic (PLEG): container finished" podID="3a059da2-0698-4630-bbba-b85e37e713f6" containerID="0dc5ed479914afed937d00d1cf0c9e2400ee04d1ae3a769982eae70c95b592ec" exitCode=0
Oct 30 19:34:49.648215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.647837    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk" event={"ID":"3a059da2-0698-4630-bbba-b85e37e713f6","Type":"ContainerDied","Data":"0dc5ed479914afed937d00d1cf0c9e2400ee04d1ae3a769982eae70c95b592ec"}
Oct 30 19:34:49.830215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.830188    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:49.959852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.959824    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e5665f-cc81-41e7-9c87-1be8941e999d-kube-api-access-lwcmk\" (UniqueName: \"kubernetes.io/projected/f0e5665f-cc81-41e7-9c87-1be8941e999d-kube-api-access-lwcmk\") pod \"f0e5665f-cc81-41e7-9c87-1be8941e999d\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") "
Oct 30 19:34:49.960043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.959885    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-xks-ca\") pod \"f0e5665f-cc81-41e7-9c87-1be8941e999d\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") "
Oct 30 19:34:49.960043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.959970    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-openshift-ca\") pod \"f0e5665f-cc81-41e7-9c87-1be8941e999d\" (UID: \"f0e5665f-cc81-41e7-9c87-1be8941e999d\") "
Oct 30 19:34:49.960329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.960302    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-openshift-ca" pod "f0e5665f-cc81-41e7-9c87-1be8941e999d" (UID: "f0e5665f-cc81-41e7-9c87-1be8941e999d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:49.960803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.960785    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:49.962271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.962242    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e5665f-cc81-41e7-9c87-1be8941e999d-kube-api-access-lwcmk" pod "f0e5665f-cc81-41e7-9c87-1be8941e999d" (UID: "f0e5665f-cc81-41e7-9c87-1be8941e999d"). InnerVolumeSpecName "kube-api-access-lwcmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:49.962363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:49.962276    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-xks-ca" pod "f0e5665f-cc81-41e7-9c87-1be8941e999d" (UID: "f0e5665f-cc81-41e7-9c87-1be8941e999d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:50.062314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:50.062237    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwcmk\" (UniqueName: \"kubernetes.io/projected/f0e5665f-cc81-41e7-9c87-1be8941e999d-kube-api-access-lwcmk\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:50.062314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:50.062267    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0e5665f-cc81-41e7-9c87-1be8941e999d-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:50.189604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:50.189043    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:50.657381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:50.657355    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"
Oct 30 19:34:50.657864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:50.657398    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2" event={"ID":"f0e5665f-cc81-41e7-9c87-1be8941e999d","Type":"ContainerDied","Data":"e59a9ccb97e9cfb35e9328206159df7ab271734bafd511472c9245fd1fc7ebfd"}
Oct 30 19:34:50.657864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:50.657446    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59a9ccb97e9cfb35e9328206159df7ab271734bafd511472c9245fd1fc7ebfd"
Oct 30 19:34:50.795322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:50.784378    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:50.899026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:50.898999    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:51.000734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.000626    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a059da2-0698-4630-bbba-b85e37e713f6-kube-api-access-f8fvm\" (UniqueName: \"kubernetes.io/projected/3a059da2-0698-4630-bbba-b85e37e713f6-kube-api-access-f8fvm\") pod \"3a059da2-0698-4630-bbba-b85e37e713f6\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") "
Oct 30 19:34:51.000734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.000677    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-openshift-ca\") pod \"3a059da2-0698-4630-bbba-b85e37e713f6\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") "
Oct 30 19:34:51.000927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.000889    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-xks-ca\") pod \"3a059da2-0698-4630-bbba-b85e37e713f6\" (UID: \"3a059da2-0698-4630-bbba-b85e37e713f6\") "
Oct 30 19:34:51.001157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.001125    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-openshift-ca" pod "3a059da2-0698-4630-bbba-b85e37e713f6" (UID: "3a059da2-0698-4630-bbba-b85e37e713f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:51.001651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.001630    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:51.003158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.003127    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-xks-ca" pod "3a059da2-0698-4630-bbba-b85e37e713f6" (UID: "3a059da2-0698-4630-bbba-b85e37e713f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:51.003274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.003214    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a059da2-0698-4630-bbba-b85e37e713f6-kube-api-access-f8fvm" pod "3a059da2-0698-4630-bbba-b85e37e713f6" (UID: "3a059da2-0698-4630-bbba-b85e37e713f6"). InnerVolumeSpecName "kube-api-access-f8fvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:51.103394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.103349    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8fvm\" (UniqueName: \"kubernetes.io/projected/3a059da2-0698-4630-bbba-b85e37e713f6-kube-api-access-f8fvm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:51.103394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.103388    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a059da2-0698-4630-bbba-b85e37e713f6-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:51.359645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.359612    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"]
Oct 30 19:34:51.359872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.359851    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy" containerID="cri-o://4f030805bd914faffacb27a3183090391f08a9360cb1085551bbb0c5ff2be747" gracePeriod=30
Oct 30 19:34:51.666581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.666553    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"
Oct 30 19:34:51.667004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.666593    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk" event={"ID":"3a059da2-0698-4630-bbba-b85e37e713f6","Type":"ContainerDied","Data":"5ec7d6b26bf2a3bc0508a71ee0d6e8c136cdfe57a051426a78840dafc27f7de2"}
Oct 30 19:34:51.667004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.666645    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ec7d6b26bf2a3bc0508a71ee0d6e8c136cdfe57a051426a78840dafc27f7de2"
Oct 30 19:34:51.761314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:51.761189    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:52.124949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:52.124875    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"]
Oct 30 19:34:52.125150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:52.125124    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy" containerID="cri-o://7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c" gracePeriod=30
Oct 30 19:34:52.129516 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:52.129488    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:34:52.694125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:52.691917    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:53.355888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:53.355852    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:54.131183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:54.131147    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:54.746587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:54.746550    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:55.400479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.400443    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"]
Oct 30 19:34:55.404858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.404828    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"]
Oct 30 19:34:55.405527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.405510    4925 cpu_manager.go:414] "containerMap: removing container" podUID="3a059da2-0698-4630-bbba-b85e37e713f6" containerName="revoke-keys"
Oct 30 19:34:55.405527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.405528    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a059da2-0698-4630-bbba-b85e37e713f6" containerName="revoke-keys"
Oct 30 19:34:55.405677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.405539    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f0e5665f-cc81-41e7-9c87-1be8941e999d" containerName="revoke-keys"
Oct 30 19:34:55.405677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.405545    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e5665f-cc81-41e7-9c87-1be8941e999d" containerName="revoke-keys"
Oct 30 19:34:55.405786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.405682    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a059da2-0698-4630-bbba-b85e37e713f6" containerName="revoke-keys"
Oct 30 19:34:55.405833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.405789    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e5665f-cc81-41e7-9c87-1be8941e999d" containerName="revoke-keys"
Oct 30 19:34:55.411203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.411184    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.416485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.416462    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-76117c-openshift-default-dockercfg-4nkjx\""
Oct 30 19:34:55.435712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.435664    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"]
Oct 30 19:34:55.462641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.462578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:55.505353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.505323    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istiod-ca-cert\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.505353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.505364    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-credential-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.505714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.505499    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-envoy\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.505714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.505524    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-podinfo\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.505714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.505649    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-certs\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.505859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.505733    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.505859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.505846    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cpkd\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-kube-api-access-7cpkd\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.505952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.505876    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-data\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.506051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.506034    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-token\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.607617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.607585    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cpkd\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-kube-api-access-7cpkd\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.607834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.607631    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-data\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.607834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.607718    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-token\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.607972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.607858    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istiod-ca-cert\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.607972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.607887    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-credential-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.608092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.607976    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-envoy\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.608092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.608003    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-podinfo\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.608182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.608087    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-certs\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.608182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.608134    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.608182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.608136    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-data\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.608320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.608266    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-credential-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.608497 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.608475    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.608639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.608561    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istiod-ca-cert\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.608747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.608723    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-certs\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.610419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.610403    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-envoy\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.610714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.610676    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-podinfo\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.619066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.619043    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-token\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.619258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.619242    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cpkd\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-kube-api-access-7cpkd\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:55.697768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.692563    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:55.759285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:55.758042    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:56.022285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.016087    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"]
Oct 30 19:34:56.162899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.162558    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:56.375743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.363739    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:56.424713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.420563    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"]
Oct 30 19:34:56.449892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.444479    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:56.449892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.444568    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:56.449892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.444612    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:34:56.670317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.670280    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:56.833713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.833357    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" event={"ID":"63c4a300-1b96-4a15-a0c2-cd8ca681129f","Type":"ContainerStarted","Data":"996e1b75021b6ff20d8e7424387faf473663e92dceab7f578f35e298e1b10203"}
Oct 30 19:34:56.833713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.833415    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" event={"ID":"63c4a300-1b96-4a15-a0c2-cd8ca681129f","Type":"ContainerStarted","Data":"3acc5a92d8d99d2a1879624dd9eb9c3cc09222a1cef5784593b4f568351183e9"}
Oct 30 19:34:56.838633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.838508    4925 generic.go:356] "Generic (PLEG): container finished" podID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerID="4f030805bd914faffacb27a3183090391f08a9360cb1085551bbb0c5ff2be747" exitCode=0
Oct 30 19:34:56.838787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:56.838732    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" event={"ID":"bcb4d366-46d4-4a62-9cb0-733a58254f2f","Type":"ContainerDied","Data":"4f030805bd914faffacb27a3183090391f08a9360cb1085551bbb0c5ff2be747"}
Oct 30 19:34:57.011311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.011281    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:34:57.039063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.039012    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podStartSLOduration=2.038997219 podStartE2EDuration="2.038997219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:56.880933534 +0000 UTC m=+1731.783946574" watchObservedRunningTime="2026-10-30 19:34:57.038997219 +0000 UTC m=+1731.942010251"
Oct 30 19:34:57.142477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.142443    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-data\") pod \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") "
Oct 30 19:34:57.142641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.142513    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-podinfo\") pod \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") "
Oct 30 19:34:57.142641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.142554    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-token\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-token\") pod \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") "
Oct 30 19:34:57.142641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.142616    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istiod-ca-cert\") pod \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") "
Oct 30 19:34:57.142840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.142740    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-certs\") pod \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") "
Oct 30 19:34:57.142840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.142834    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-envoy\") pod \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") "
Oct 30 19:34:57.143040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.143025    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-kube-api-access-n85dn\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-kube-api-access-n85dn\") pod \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") "
Oct 30 19:34:57.143100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.143075    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-credential-socket\") pod \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") "
Oct 30 19:34:57.143172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.143160    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-socket\") pod \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\" (UID: \"bcb4d366-46d4-4a62-9cb0-733a58254f2f\") "
Oct 30 19:34:57.145024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.144916    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-data" pod "bcb4d366-46d4-4a62-9cb0-733a58254f2f" (UID: "bcb4d366-46d4-4a62-9cb0-733a58254f2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.145168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.145139    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-certs" pod "bcb4d366-46d4-4a62-9cb0-733a58254f2f" (UID: "bcb4d366-46d4-4a62-9cb0-733a58254f2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.145353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.145150    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istiod-ca-cert" pod "bcb4d366-46d4-4a62-9cb0-733a58254f2f" (UID: "bcb4d366-46d4-4a62-9cb0-733a58254f2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:57.181719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.178139    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-credential-socket" pod "bcb4d366-46d4-4a62-9cb0-733a58254f2f" (UID: "bcb4d366-46d4-4a62-9cb0-733a58254f2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.181719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.178371    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-socket" pod "bcb4d366-46d4-4a62-9cb0-733a58254f2f" (UID: "bcb4d366-46d4-4a62-9cb0-733a58254f2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.185975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.183386    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-podinfo" pod "bcb4d366-46d4-4a62-9cb0-733a58254f2f" (UID: "bcb4d366-46d4-4a62-9cb0-733a58254f2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:57.192901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.190839    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-envoy" pod "bcb4d366-46d4-4a62-9cb0-733a58254f2f" (UID: "bcb4d366-46d4-4a62-9cb0-733a58254f2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.192901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.192019    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-token" pod "bcb4d366-46d4-4a62-9cb0-733a58254f2f" (UID: "bcb4d366-46d4-4a62-9cb0-733a58254f2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:57.193344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.193297    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-kube-api-access-n85dn" pod "bcb4d366-46d4-4a62-9cb0-733a58254f2f" (UID: "bcb4d366-46d4-4a62-9cb0-733a58254f2f"). InnerVolumeSpecName "kube-api-access-n85dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:57.245822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.245789    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.245822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.245811    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.245822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.245820    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.245951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.245831    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n85dn\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-kube-api-access-n85dn\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.245951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.245841    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.245951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.245849    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.245951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.245857    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.245951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.245865    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.245951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.245873    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb4d366-46d4-4a62-9cb0-733a58254f2f-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.350362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.350268    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:57.450907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.450879    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:34:57.527332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.527186    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:57.550649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.550614    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-envoy\") pod \"d9215648-6e52-454f-b9cd-1e2985794e9f\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") "
Oct 30 19:34:57.550914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.550806    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-kube-api-access-cb9hq\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-kube-api-access-cb9hq\") pod \"d9215648-6e52-454f-b9cd-1e2985794e9f\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") "
Oct 30 19:34:57.551004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.550914    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-podinfo\") pod \"d9215648-6e52-454f-b9cd-1e2985794e9f\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") "
Oct 30 19:34:57.551148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.551124    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9215648-6e52-454f-b9cd-1e2985794e9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9215648-6e52-454f-b9cd-1e2985794e9f-istiod-ca-cert\") pod \"d9215648-6e52-454f-b9cd-1e2985794e9f\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") "
Oct 30 19:34:57.551228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.551176    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-credential-socket\") pod \"d9215648-6e52-454f-b9cd-1e2985794e9f\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") "
Oct 30 19:34:57.551308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.551277    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-socket\") pod \"d9215648-6e52-454f-b9cd-1e2985794e9f\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") "
Oct 30 19:34:57.551379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.551362    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-token\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-token\") pod \"d9215648-6e52-454f-b9cd-1e2985794e9f\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") "
Oct 30 19:34:57.551537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.551442    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-data\") pod \"d9215648-6e52-454f-b9cd-1e2985794e9f\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") "
Oct 30 19:34:57.551706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.551611    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-certs\") pod \"d9215648-6e52-454f-b9cd-1e2985794e9f\" (UID: \"d9215648-6e52-454f-b9cd-1e2985794e9f\") "
Oct 30 19:34:57.552231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.551995    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-credential-socket" pod "d9215648-6e52-454f-b9cd-1e2985794e9f" (UID: "d9215648-6e52-454f-b9cd-1e2985794e9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.552334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.552280    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-certs" pod "d9215648-6e52-454f-b9cd-1e2985794e9f" (UID: "d9215648-6e52-454f-b9cd-1e2985794e9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.552968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.552496    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-socket" pod "d9215648-6e52-454f-b9cd-1e2985794e9f" (UID: "d9215648-6e52-454f-b9cd-1e2985794e9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.555365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.555145    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.555365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.555183    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.555365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.555200    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.571712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.556284    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9215648-6e52-454f-b9cd-1e2985794e9f-istiod-ca-cert" pod "d9215648-6e52-454f-b9cd-1e2985794e9f" (UID: "d9215648-6e52-454f-b9cd-1e2985794e9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:57.571712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.553065    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-data" pod "d9215648-6e52-454f-b9cd-1e2985794e9f" (UID: "d9215648-6e52-454f-b9cd-1e2985794e9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.591877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.591829    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-envoy" pod "d9215648-6e52-454f-b9cd-1e2985794e9f" (UID: "d9215648-6e52-454f-b9cd-1e2985794e9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:57.591877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.591871    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-kube-api-access-cb9hq" pod "d9215648-6e52-454f-b9cd-1e2985794e9f" (UID: "d9215648-6e52-454f-b9cd-1e2985794e9f"). InnerVolumeSpecName "kube-api-access-cb9hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:57.594550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.594527    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"]
Oct 30 19:34:57.595083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.595045    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy" containerID="cri-o://fb9b40aa9f8acd05216f2bad926a481b15dcffe97505042354aa13772daa8efa" gracePeriod=30
Oct 30 19:34:57.612937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.611978    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-token" pod "d9215648-6e52-454f-b9cd-1e2985794e9f" (UID: "d9215648-6e52-454f-b9cd-1e2985794e9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:57.612937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.612167    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-podinfo" pod "d9215648-6e52-454f-b9cd-1e2985794e9f" (UID: "d9215648-6e52-454f-b9cd-1e2985794e9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:57.657101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.656901    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb9hq\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-kube-api-access-cb9hq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.657101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.656936    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.657101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.656952    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9215648-6e52-454f-b9cd-1e2985794e9f-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.657101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.656966    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.657101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.656980    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.657101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.656994    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9215648-6e52-454f-b9cd-1e2985794e9f-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:57.801140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.762458    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:34:57.845546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.845509    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:57.877885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.849916    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 19:34:57.878400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:57.878132    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 19:34:58.141778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.141459    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs" event={"ID":"bcb4d366-46d4-4a62-9cb0-733a58254f2f","Type":"ContainerDied","Data":"3442e7955b129d20f6d6ecc21ef1e956ed3097cece24bd779c122e45b88bb956"}
Oct 30 19:34:58.141778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.141525    4925 scope.go:122] "RemoveContainer" containerID="4f030805bd914faffacb27a3183090391f08a9360cb1085551bbb0c5ff2be747"
Oct 30 19:34:58.141778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.141712    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"
Oct 30 19:34:58.150184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.150154    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:58.173362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.171047    4925 generic.go:356] "Generic (PLEG): container finished" podID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerID="7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c" exitCode=0
Oct 30 19:34:58.173362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.171758    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"
Oct 30 19:34:58.173362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.172765    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" event={"ID":"d9215648-6e52-454f-b9cd-1e2985794e9f","Type":"ContainerDied","Data":"7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c"}
Oct 30 19:34:58.173362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.172814    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw" event={"ID":"d9215648-6e52-454f-b9cd-1e2985794e9f","Type":"ContainerDied","Data":"7982bcc6e0d602d0800b64f27bd5fc6741a65b05617227c54582bd87cbe2dfa5"}
Oct 30 19:34:58.248644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.248611    4925 scope.go:122] "RemoveContainer" containerID="7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c"
Oct 30 19:34:58.255588 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.252732    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"]
Oct 30 19:34:58.259063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.259041    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-g8hrs"]
Oct 30 19:34:58.264071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.264051    4925 scope.go:122] "RemoveContainer" containerID="7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c"
Oct 30 19:34:58.264938 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:34:58.264918    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c\": container with ID starting with 7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c not found: ID does not exist" containerID="7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c"
Oct 30 19:34:58.265014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.264947    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c"} err="failed to get container status \"7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c\": rpc error: code = NotFound desc = could not find container \"7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c\": container with ID starting with 7e88e6cfbecb785091a0e8eb48abd0c131d29418f0d7a9e3e1d5b5de520c810c not found: ID does not exist"
Oct 30 19:34:58.268190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.268170    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"]
Oct 30 19:34:58.277484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.277462    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-c7jqw"]
Oct 30 19:34:58.365028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.364996    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:58.551246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.551146    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:58.759581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.759546    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 19:34:58.759763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:58.759595    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 19:34:59.086304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.086269    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:59.146719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.107782    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"]
Oct 30 19:34:59.146719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.108870    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy"
Oct 30 19:34:59.146719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.108891    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy"
Oct 30 19:34:59.146719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.108950    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy"
Oct 30 19:34:59.146719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.108960    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy"
Oct 30 19:34:59.146719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.109232    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" containerName="istio-proxy"
Oct 30 19:34:59.146719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.109250    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" containerName="istio-proxy"
Oct 30 19:34:59.377766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.375525    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.534749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.533393    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ddb6ea-openshift-default-dockercfg-z5667\""
Oct 30 19:34:59.575775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.562823    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:59.619985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.619357    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d846076-2d4e-42fe-8909-b6fde688f282-istio-podinfo\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.619985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.619457    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-istio-token\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.619985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.619513    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-data\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.619985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.619573    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d846076-2d4e-42fe-8909-b6fde688f282-istiod-ca-cert\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.619985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.619622    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.619985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.619679    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjwg\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-kube-api-access-lzjwg\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.619985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.619778    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-credential-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.619985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.619846    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-envoy\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.636651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.636266    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-certs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.671644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.671611    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb4d366-46d4-4a62-9cb0-733a58254f2f" path="/var/lib/kubelet/pods/bcb4d366-46d4-4a62-9cb0-733a58254f2f/volumes"
Oct 30 19:34:59.672724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.672674    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9215648-6e52-454f-b9cd-1e2985794e9f" path="/var/lib/kubelet/pods/d9215648-6e52-454f-b9cd-1e2985794e9f/volumes"
Oct 30 19:34:59.673454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.673432    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"]
Oct 30 19:34:59.741658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.741621    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-certs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.741877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.741843    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d846076-2d4e-42fe-8909-b6fde688f282-istio-podinfo\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.741930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.741913    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-istio-token\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.741988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.741967    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-data\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.742040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.742026    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d846076-2d4e-42fe-8909-b6fde688f282-istiod-ca-cert\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.742093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.742074    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.742155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.742108    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-certs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.742155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.742128    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjwg\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-kube-api-access-lzjwg\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.742264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.742192    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-credential-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.742264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.742243    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-envoy\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.742525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.742504    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-data\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.742866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.742840    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d846076-2d4e-42fe-8909-b6fde688f282-istiod-ca-cert\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.742967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.742945    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.743040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.743019    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-credential-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.744893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.744870    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-envoy\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.746881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.746858    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:59.747216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.747194    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d846076-2d4e-42fe-8909-b6fde688f282-istio-podinfo\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.754364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.754340    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjwg\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-kube-api-access-lzjwg\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.754747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.754729    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-istio-token\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:34:59.760054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.760030    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 19:34:59.760758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.760077    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 19:34:59.965916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:34:59.965888    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:35:00.255746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.255641    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:00.329780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.324015    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"]
Oct 30 19:35:00.405043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.391191    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:00.406926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.406888    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:00.407158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.407131    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:00.407772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.405313    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh" podUID="f08e19cf-765f-46fc-997c-c25e6a7dae3a" containerName="payload-processing" containerID="cri-o://1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1" gracePeriod=25
Oct 30 19:35:00.428014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.427983    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:00.693299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.693263    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" event={"ID":"5d846076-2d4e-42fe-8909-b6fde688f282","Type":"ContainerStarted","Data":"b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3"}
Oct 30 19:35:00.693704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.693329    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" event={"ID":"5d846076-2d4e-42fe-8909-b6fde688f282","Type":"ContainerStarted","Data":"f787779dc573826652e29562e12bfdb900b84d3fb651a5982af91f65a59d8276"}
Oct 30 19:35:00.759712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.756919    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q" podUID="7868c7ae-6114-40b4-9d1c-68e85e127437" containerName="main" containerID="cri-o://d384a823f573621c3f99c59838303777c22f33d95828ff4a0ddb0d3564e18b56" gracePeriod=45
Oct 30 19:35:00.766575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.766247    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 19:35:00.766575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.766301    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 19:35:00.771265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.770313    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" podStartSLOduration=1.7702936870000001 podStartE2EDuration="1.770293687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:00.742062633 +0000 UTC m=+1735.645075672" watchObservedRunningTime="2026-10-30 19:35:00.770293687 +0000 UTC m=+1735.673306731"
Oct 30 19:35:00.912648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.895864    4925 patch_prober.go:29] interesting pod/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:35:00.912648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.895918    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:35:00.944774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.944104    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q" podUID="7868c7ae-6114-40b4-9d1c-68e85e127437" containerName="main" probeResult="failure" output="Get \"https://10.132.0.155:8000/ready\": dial tcp 10.132.0.155:8000: connect: connection refused"
Oct 30 19:35:00.968716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:00.966915    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:35:01.034875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.034776    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq" podUID="2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" containerName="payload-processing" containerID="cri-o://7b204e6d5f944e5b23cf5b1019624330ba38ddde820aa36926ef9fbfb6ed2b5f" gracePeriod=25
Oct 30 19:35:01.198718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.195383    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:01.756576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.756545    4925 generic.go:356] "Generic (PLEG): container finished" podID="7868c7ae-6114-40b4-9d1c-68e85e127437" containerID="d384a823f573621c3f99c59838303777c22f33d95828ff4a0ddb0d3564e18b56" exitCode=0
Oct 30 19:35:01.758758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.758736    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q" event={"ID":"7868c7ae-6114-40b4-9d1c-68e85e127437","Type":"ContainerDied","Data":"d384a823f573621c3f99c59838303777c22f33d95828ff4a0ddb0d3564e18b56"}
Oct 30 19:35:01.760507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.760484    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 19:35:01.760656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.760626    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 19:35:01.794040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.794010    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:35:01.887643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.887611    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7868c7ae-6114-40b4-9d1c-68e85e127437-tls-certs\" (UniqueName: \"kubernetes.io/secret/7868c7ae-6114-40b4-9d1c-68e85e127437-tls-certs\") pod \"7868c7ae-6114-40b4-9d1c-68e85e127437\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") "
Oct 30 19:35:01.887827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.887758    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7868c7ae-6114-40b4-9d1c-68e85e127437-kube-api-access-m4qzt\" (UniqueName: \"kubernetes.io/projected/7868c7ae-6114-40b4-9d1c-68e85e127437-kube-api-access-m4qzt\") pod \"7868c7ae-6114-40b4-9d1c-68e85e127437\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") "
Oct 30 19:35:01.887827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.887821    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-model-cache\") pod \"7868c7ae-6114-40b4-9d1c-68e85e127437\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") "
Oct 30 19:35:01.888000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.887984    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-tmp-dir\") pod \"7868c7ae-6114-40b4-9d1c-68e85e127437\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") "
Oct 30 19:35:01.888124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.888107    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-home\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-home\") pod \"7868c7ae-6114-40b4-9d1c-68e85e127437\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") "
Oct 30 19:35:01.888277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.888116    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-model-cache" pod "7868c7ae-6114-40b4-9d1c-68e85e127437" (UID: "7868c7ae-6114-40b4-9d1c-68e85e127437"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:01.888277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.888204    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-tmp-dir" pod "7868c7ae-6114-40b4-9d1c-68e85e127437" (UID: "7868c7ae-6114-40b4-9d1c-68e85e127437"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:01.888277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.888221    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-dshm\") pod \"7868c7ae-6114-40b4-9d1c-68e85e127437\" (UID: \"7868c7ae-6114-40b4-9d1c-68e85e127437\") "
Oct 30 19:35:01.888455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.888304    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-home" pod "7868c7ae-6114-40b4-9d1c-68e85e127437" (UID: "7868c7ae-6114-40b4-9d1c-68e85e127437"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:01.888974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.888951    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.889080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.888976    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.889080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.888989    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.890316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.890291    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7868c7ae-6114-40b4-9d1c-68e85e127437-kube-api-access-m4qzt" pod "7868c7ae-6114-40b4-9d1c-68e85e127437" (UID: "7868c7ae-6114-40b4-9d1c-68e85e127437"). InnerVolumeSpecName "kube-api-access-m4qzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:01.890401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.890312    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-dshm" pod "7868c7ae-6114-40b4-9d1c-68e85e127437" (UID: "7868c7ae-6114-40b4-9d1c-68e85e127437"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:01.890595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.890570    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7868c7ae-6114-40b4-9d1c-68e85e127437-tls-certs" pod "7868c7ae-6114-40b4-9d1c-68e85e127437" (UID: "7868c7ae-6114-40b4-9d1c-68e85e127437"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:01.973551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.973512    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:35:01.990490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.990453    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7868c7ae-6114-40b4-9d1c-68e85e127437-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.990490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.990479    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4qzt\" (UniqueName: \"kubernetes.io/projected/7868c7ae-6114-40b4-9d1c-68e85e127437-kube-api-access-m4qzt\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.990490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:01.990490    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7868c7ae-6114-40b4-9d1c-68e85e127437-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:02.170298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.168382    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:02.761520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.761483    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 19:35:02.762051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.761541    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 19:35:02.777734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.775777    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q" event={"ID":"7868c7ae-6114-40b4-9d1c-68e85e127437","Type":"ContainerDied","Data":"dc6306fcc0d27b52e8cd87183fd5a9df7279a6e2bf9525f494ceeeeb57fa94ca"}
Oct 30 19:35:02.777734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.775819    4925 scope.go:122] "RemoveContainer" containerID="d384a823f573621c3f99c59838303777c22f33d95828ff4a0ddb0d3564e18b56"
Oct 30 19:35:02.777734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.775925    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"
Oct 30 19:35:02.781864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.781827    4925 generic.go:356] "Generic (PLEG): container finished" podID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerID="fb9b40aa9f8acd05216f2bad926a481b15dcffe97505042354aa13772daa8efa" exitCode=0
Oct 30 19:35:02.782816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.782791    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" event={"ID":"fb24c7dd-850b-40a4-b25b-2450e5fffe39","Type":"ContainerDied","Data":"fb9b40aa9f8acd05216f2bad926a481b15dcffe97505042354aa13772daa8efa"}
Oct 30 19:35:02.783382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.783137    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:35:02.784375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.784354    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:35:02.832701 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.832658    4925 scope.go:122] "RemoveContainer" containerID="eacfe6ef81a7811d3b8141d985b42a866e9dcccf04200f89c89022dd13d2dce1"
Oct 30 19:35:02.855799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.855772    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"]
Oct 30 19:35:02.859216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.859161    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c86696/test-model-c86696-kserve-54f84db5bc-plx5q"]
Oct 30 19:35:02.893644 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:02.893618    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eacfe6ef81a7811d3b8141d985b42a866e9dcccf04200f89c89022dd13d2dce1\": container with ID starting with eacfe6ef81a7811d3b8141d985b42a866e9dcccf04200f89c89022dd13d2dce1 not found: ID does not exist" containerID="eacfe6ef81a7811d3b8141d985b42a866e9dcccf04200f89c89022dd13d2dce1"
Oct 30 19:35:02.907906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:02.907874    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.045645    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-envoy\") pod \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") "
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.045872    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-kube-api-access-xpkw9\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-kube-api-access-xpkw9\") pod \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") "
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.045935    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-token\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-token\") pod \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") "
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.046037    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istiod-ca-cert\") pod \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") "
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.046106    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-credential-socket\") pod \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") "
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.046248    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-podinfo\") pod \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") "
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.046335    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-data\") pod \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") "
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.046402    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-socket\") pod \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") "
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.046448    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-certs\") pod \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\" (UID: \"fb24c7dd-850b-40a4-b25b-2450e5fffe39\") "
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.049130    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istiod-ca-cert" pod "fb24c7dd-850b-40a4-b25b-2450e5fffe39" (UID: "fb24c7dd-850b-40a4-b25b-2450e5fffe39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:03.053013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.052540    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-credential-socket" pod "fb24c7dd-850b-40a4-b25b-2450e5fffe39" (UID: "fb24c7dd-850b-40a4-b25b-2450e5fffe39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.077480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.069517    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-socket" pod "fb24c7dd-850b-40a4-b25b-2450e5fffe39" (UID: "fb24c7dd-850b-40a4-b25b-2450e5fffe39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.077480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.072897    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:03.077480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.073541    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-envoy" pod "fb24c7dd-850b-40a4-b25b-2450e5fffe39" (UID: "fb24c7dd-850b-40a4-b25b-2450e5fffe39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.077480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.073672    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-token" pod "fb24c7dd-850b-40a4-b25b-2450e5fffe39" (UID: "fb24c7dd-850b-40a4-b25b-2450e5fffe39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:03.077480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.073886    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-certs" pod "fb24c7dd-850b-40a4-b25b-2450e5fffe39" (UID: "fb24c7dd-850b-40a4-b25b-2450e5fffe39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.077480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.076772    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-data" pod "fb24c7dd-850b-40a4-b25b-2450e5fffe39" (UID: "fb24c7dd-850b-40a4-b25b-2450e5fffe39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:03.078280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.078211    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-podinfo" pod "fb24c7dd-850b-40a4-b25b-2450e5fffe39" (UID: "fb24c7dd-850b-40a4-b25b-2450e5fffe39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:03.078363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.078341    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-kube-api-access-xpkw9" pod "fb24c7dd-850b-40a4-b25b-2450e5fffe39" (UID: "fb24c7dd-850b-40a4-b25b-2450e5fffe39"). InnerVolumeSpecName "kube-api-access-xpkw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:03.079831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.079518    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq" podUID="2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:03.151313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.151227    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.151313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.151261    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.151313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.151271    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.151313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.151280    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.151313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.151289    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.151313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.151300    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpkw9\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-kube-api-access-xpkw9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.151313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.151311    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.151313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.151319    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb24c7dd-850b-40a4-b25b-2450e5fffe39-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.151656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.151327    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb24c7dd-850b-40a4-b25b-2450e5fffe39-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:03.265200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.265169    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7868c7ae-6114-40b4-9d1c-68e85e127437" path="/var/lib/kubelet/pods/7868c7ae-6114-40b4-9d1c-68e85e127437/volumes"
Oct 30 19:35:03.274666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.274641    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:03.479561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.479475    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:03.759747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.759650    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 30 19:35:03.759747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.759737    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 30 19:35:03.792902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.792873    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"
Oct 30 19:35:03.793400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.792910    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p" event={"ID":"fb24c7dd-850b-40a4-b25b-2450e5fffe39","Type":"ContainerDied","Data":"aa288202ca014c5e6143068f27e0e38b099bb6bb283328176916c08d6d1e0975"}
Oct 30 19:35:03.793400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.792973    4925 scope.go:122] "RemoveContainer" containerID="fb9b40aa9f8acd05216f2bad926a481b15dcffe97505042354aa13772daa8efa"
Oct 30 19:35:03.847145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.847110    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"]
Oct 30 19:35:03.853057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:03.853032    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-afdf16-openshift-default-5c66498b5c-q886p"]
Oct 30 19:35:04.126282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.126189    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"]
Oct 30 19:35:04.127142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.127119    4925 cpu_manager.go:414] "containerMap: removing container" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy"
Oct 30 19:35:04.127142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.127142    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy"
Oct 30 19:35:04.127323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.127168    4925 cpu_manager.go:414] "containerMap: removing container" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy"
Oct 30 19:35:04.127323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.127175    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy"
Oct 30 19:35:04.127323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.127206    4925 cpu_manager.go:414] "containerMap: removing container" podUID="7868c7ae-6114-40b4-9d1c-68e85e127437" containerName="main"
Oct 30 19:35:04.127323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.127213    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7868c7ae-6114-40b4-9d1c-68e85e127437" containerName="main"
Oct 30 19:35:04.127518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.127389    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy"
Oct 30 19:35:04.127518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.127407    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" containerName="istio-proxy"
Oct 30 19:35:04.127518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.127423    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7868c7ae-6114-40b4-9d1c-68e85e127437" containerName="main"
Oct 30 19:35:04.132764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.132738    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.227605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.227276    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c24c01-openshift-default-dockercfg-6xcnv\""
Oct 30 19:35:04.241809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.241604    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:04.244298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.244273    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"]
Oct 30 19:35:04.271204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.271167    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-token\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.271204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.271204    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-data\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.271436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.271411    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-certs\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.271495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.271472    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-podinfo\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.271634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.271618    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwf8v\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-kube-api-access-wwf8v\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.271801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.271780    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.271849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.271810    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istiod-ca-cert\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.271899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.271849    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-credential-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.271981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.271962    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-envoy\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.373396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.373358    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwf8v\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-kube-api-access-wwf8v\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.373794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.373768    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.373920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.373805    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istiod-ca-cert\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.373920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.373851    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-credential-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.373920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.373914    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-envoy\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.374078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.373953    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-token\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.374078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.373972    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-data\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.374078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.374055    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-certs\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.374290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.374100    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-podinfo\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.374290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.374191    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-credential-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.374290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.374214    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-socket\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.374644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.374617    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istiod-ca-cert\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.374794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.374743    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-certs\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.374912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.374890    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-data\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.376493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.376437    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-envoy\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.376828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.376810    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-podinfo\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.394632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.394609    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-token\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.394806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.394750    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwf8v\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-kube-api-access-wwf8v\") pod \"e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") " pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.535336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.535306    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:04.600748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.600666    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh" podUID="f08e19cf-765f-46fc-997c-c25e6a7dae3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:04.665425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.665327    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:04.958264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:04.958105    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:05.014094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.014047    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"]
Oct 30 19:35:05.089223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.089164    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:05.089390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.089273    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:05.089390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.089317    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:05.269162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.269073    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb24c7dd-850b-40a4-b25b-2450e5fffe39" path="/var/lib/kubelet/pods/fb24c7dd-850b-40a4-b25b-2450e5fffe39/volumes"
Oct 30 19:35:05.293016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.292985    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:35:05.293518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.293493    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:35:05.294440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.294422    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:35:05.447083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.447048    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:05.977658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.977624    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" event={"ID":"304ce70e-b48b-4ab2-bdc7-ee6206e3143d","Type":"ContainerStarted","Data":"f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134"}
Oct 30 19:35:05.978089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:05.977708    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" event={"ID":"304ce70e-b48b-4ab2-bdc7-ee6206e3143d","Type":"ContainerStarted","Data":"b68003503461c02cf3eea07432010732e0a94b1fdaa7ea3af7e5cc49c51b405a"}
Oct 30 19:35:06.187725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:06.187676    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:06.537010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:06.536223    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:06.537750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:06.537721    4925 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:35:06.537881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:06.537782    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 19:35:06.855017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:06.854915    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:07.074712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:07.073733    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:07.369724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:07.366126    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:07.535853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:07.535819    4925 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:35:07.536112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:07.535874    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 19:35:07.694035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:07.694000    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:08.331714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:08.324416    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:08.536722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:08.536657    4925 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:35:08.536904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:08.536759    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 19:35:09.168725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:09.168699    4925 scope.go:122] "RemoveContainer" containerID="8adced0753e760b392cf400eb0f37499535e5100b5290c84ce101df8605b4b6d"
Oct 30 19:35:09.180050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:09.180033    4925 scope.go:122] "RemoveContainer" containerID="3e9f6f88db228275820f17c8ba0ceaed00a22ce50e00ac67b9a1b9ee0046bd1e"
Oct 30 19:35:09.226718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:09.226358    4925 scope.go:122] "RemoveContainer" containerID="51a33b780f043bacdca2857f735ef31f37ff3e54e90f0668e1bc558986a34a0d"
Oct 30 19:35:09.286446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:09.286414    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:09.441992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:09.441454    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" podStartSLOduration=5.441434224 podStartE2EDuration="5.441434224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:06.021354124 +0000 UTC m=+1740.924367162" watchObservedRunningTime="2026-10-30 19:35:09.441434224 +0000 UTC m=+1744.344447263"
Oct 30 19:35:09.449199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:09.449166    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"]
Oct 30 19:35:09.548084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:09.539092    4925 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:35:09.548084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:09.539171    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 19:35:09.650217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:09.644816    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:10.282676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:10.282646    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:10.535953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:10.535866    4925 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:35:10.535953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:10.535922    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 19:35:10.933074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:10.933031    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:11.216716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:11.212171    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:11.419713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:11.402030    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:11.514769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:11.514605    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"]
Oct 30 19:35:11.537742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:11.536182    4925 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:35:11.537742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:11.536240    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 19:35:11.555717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:11.555167    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-02feaa-p4zh2"]
Oct 30 19:35:11.648543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:11.648512    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:12.287271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:12.287214    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:12.542644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:12.542568    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:12.544230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:12.544207    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:12.544785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:12.544763    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:35:12.856116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:12.856033    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:13.055218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:13.051932    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:13.078516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:13.078478    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq" podUID="2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:13.237447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:13.237413    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:13.264788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:13.264759    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e5665f-cc81-41e7-9c87-1be8941e999d" path="/var/lib/kubelet/pods/f0e5665f-cc81-41e7-9c87-1be8941e999d/volumes"
Oct 30 19:35:13.972350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:13.972317    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:14.599937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:14.599899    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh" podUID="f08e19cf-765f-46fc-997c-c25e6a7dae3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:14.911110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:14.911075    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:15.581235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:15.581183    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:15.873852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:15.873741    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:16.043728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:16.043610    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:16.195572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:16.195469    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"]
Oct 30 19:35:16.196173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:16.195736    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy" containerID="cri-o://0c2d65904d793c8509695edead5f402e63a34f4ed61f7109d43cd1cefba2a4c5" gracePeriod=30
Oct 30 19:35:16.202501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:16.202474    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"]
Oct 30 19:35:16.202745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:16.202720    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy" containerID="cri-o://a2c95115997cb98f1c0da1c64a9a2b72d0e545615fba8e3af0a14ce45dcbdcd0" gracePeriod=30
Oct 30 19:35:16.338716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:16.337742    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:16.524770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:16.523979    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:17.216899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:17.216863    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:18.242145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:18.242096    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:19.371392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:19.366582    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:19.580864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:19.580825    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:19.756839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:19.756797    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:19.881793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:19.881761    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"]
Oct 30 19:35:19.882072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:19.882049    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy" containerID="cri-o://581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383" gracePeriod=30
Oct 30 19:35:19.885645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:19.885614    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"]
Oct 30 19:35:19.885910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:19.885882    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy" containerID="cri-o://bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c" gracePeriod=30
Oct 30 19:35:19.953650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:19.953621    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:20.153715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:20.150149    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:20.655708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:20.655667    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:35:20.669195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:20.669167    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08e19cf-765f-46fc-997c-c25e6a7dae3a-kube-api-access-llk6x\" (UniqueName: \"kubernetes.io/projected/f08e19cf-765f-46fc-997c-c25e6a7dae3a-kube-api-access-llk6x\") pod \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\" (UID: \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\") "
Oct 30 19:35:20.669326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:20.669221    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08e19cf-765f-46fc-997c-c25e6a7dae3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08e19cf-765f-46fc-997c-c25e6a7dae3a-plugins-config-volume\") pod \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\" (UID: \"f08e19cf-765f-46fc-997c-c25e6a7dae3a\") "
Oct 30 19:35:20.669653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:20.669630    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08e19cf-765f-46fc-997c-c25e6a7dae3a-plugins-config-volume" pod "f08e19cf-765f-46fc-997c-c25e6a7dae3a" (UID: "f08e19cf-765f-46fc-997c-c25e6a7dae3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:20.670057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:20.670040    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08e19cf-765f-46fc-997c-c25e6a7dae3a-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:20.671397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:20.671377    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08e19cf-765f-46fc-997c-c25e6a7dae3a-kube-api-access-llk6x" pod "f08e19cf-765f-46fc-997c-c25e6a7dae3a" (UID: "f08e19cf-765f-46fc-997c-c25e6a7dae3a"). InnerVolumeSpecName "kube-api-access-llk6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:20.771478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:20.771445    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llk6x\" (UniqueName: \"kubernetes.io/projected/f08e19cf-765f-46fc-997c-c25e6a7dae3a-kube-api-access-llk6x\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.044123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.044032    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:21.393009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.392967    4925 generic.go:356] "Generic (PLEG): container finished" podID="2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" containerID="7b204e6d5f944e5b23cf5b1019624330ba38ddde820aa36926ef9fbfb6ed2b5f" exitCode=0
Oct 30 19:35:21.393196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.393062    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq" event={"ID":"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d","Type":"ContainerDied","Data":"7b204e6d5f944e5b23cf5b1019624330ba38ddde820aa36926ef9fbfb6ed2b5f"}
Oct 30 19:35:21.398722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.394722    4925 generic.go:356] "Generic (PLEG): container finished" podID="f08e19cf-765f-46fc-997c-c25e6a7dae3a" containerID="1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1" exitCode=0
Oct 30 19:35:21.398722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.394814    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh" event={"ID":"f08e19cf-765f-46fc-997c-c25e6a7dae3a","Type":"ContainerDied","Data":"1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1"}
Oct 30 19:35:21.398722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.394871    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"
Oct 30 19:35:21.398722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.394888    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh" event={"ID":"f08e19cf-765f-46fc-997c-c25e6a7dae3a","Type":"ContainerDied","Data":"571cf12bea8c2ed6a195931fde554c762dc1840a2dfcfe2e4d430fdb110a05bf"}
Oct 30 19:35:21.398722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.394921    4925 scope.go:122] "RemoveContainer" containerID="1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1"
Oct 30 19:35:21.398722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.397278    4925 generic.go:356] "Generic (PLEG): container finished" podID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerID="a2c95115997cb98f1c0da1c64a9a2b72d0e545615fba8e3af0a14ce45dcbdcd0" exitCode=0
Oct 30 19:35:21.398722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.397647    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" event={"ID":"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48","Type":"ContainerDied","Data":"a2c95115997cb98f1c0da1c64a9a2b72d0e545615fba8e3af0a14ce45dcbdcd0"}
Oct 30 19:35:21.401570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.401518    4925 generic.go:356] "Generic (PLEG): container finished" podID="af7a4772-c323-4771-b082-59a9510df9f8" containerID="0c2d65904d793c8509695edead5f402e63a34f4ed61f7109d43cd1cefba2a4c5" exitCode=0
Oct 30 19:35:21.401838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.401596    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" event={"ID":"af7a4772-c323-4771-b082-59a9510df9f8","Type":"ContainerDied","Data":"0c2d65904d793c8509695edead5f402e63a34f4ed61f7109d43cd1cefba2a4c5"}
Oct 30 19:35:21.454905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.454668    4925 scope.go:122] "RemoveContainer" containerID="1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1"
Oct 30 19:35:21.455077 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:21.455047    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1\": container with ID starting with 1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1 not found: ID does not exist" containerID="1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1"
Oct 30 19:35:21.455236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.455092    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1"} err="failed to get container status \"1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1\": rpc error: code = NotFound desc = could not find container \"1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1\": container with ID starting with 1cfe5581d0774921d968bb1f81fa6ab8018b3c56e3744684467bc039a79e30f1 not found: ID does not exist"
Oct 30 19:35:21.455236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.455120    4925 scope.go:122] "RemoveContainer" containerID="fe72b2c7b41d8936b0d601440861546412af3f353659c6955d0c312ebef8fa8f"
Oct 30 19:35:21.455236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.455223    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"]
Oct 30 19:35:21.462869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.461227    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-02feaa-5db7cd8cd-vq8rh"]
Oct 30 19:35:21.642488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.642376    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:35:21.697205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.697174    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-plugins-config-volume\") pod \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\" (UID: \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\") "
Oct 30 19:35:21.697550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.697387    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-kube-api-access-776wg\" (UniqueName: \"kubernetes.io/projected/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-kube-api-access-776wg\") pod \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\" (UID: \"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d\") "
Oct 30 19:35:21.697637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.697609    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-plugins-config-volume" pod "2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" (UID: "2d159211-5fdd-4e5f-8e50-034d2a4e3f4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:21.699536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.699507    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.700033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.700011    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-kube-api-access-776wg" pod "2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" (UID: "2d159211-5fdd-4e5f-8e50-034d2a4e3f4d"). InnerVolumeSpecName "kube-api-access-776wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:21.702219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.702200    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:35:21.708065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.708047    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:35:21.800994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.800955    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af7a4772-c323-4771-b082-59a9510df9f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7a4772-c323-4771-b082-59a9510df9f8-istio-podinfo\") pod \"af7a4772-c323-4771-b082-59a9510df9f8\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") "
Oct 30 19:35:21.801188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801039    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-envoy\") pod \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") "
Oct 30 19:35:21.801188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801139    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-data\") pod \"af7a4772-c323-4771-b082-59a9510df9f8\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") "
Oct 30 19:35:21.801188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801170    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istiod-ca-cert\") pod \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") "
Oct 30 19:35:21.801403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801219    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-kube-api-access-4md6t\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-kube-api-access-4md6t\") pod \"af7a4772-c323-4771-b082-59a9510df9f8\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") "
Oct 30 19:35:21.801403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801290    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-socket\") pod \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") "
Oct 30 19:35:21.801403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801359    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-certs\") pod \"af7a4772-c323-4771-b082-59a9510df9f8\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") "
Oct 30 19:35:21.801403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801394    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-istio-token\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-istio-token\") pod \"af7a4772-c323-4771-b082-59a9510df9f8\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") "
Oct 30 19:35:21.801757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801459    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-envoy\") pod \"af7a4772-c323-4771-b082-59a9510df9f8\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") "
Oct 30 19:35:21.801757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801492    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af7a4772-c323-4771-b082-59a9510df9f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7a4772-c323-4771-b082-59a9510df9f8-istiod-ca-cert\") pod \"af7a4772-c323-4771-b082-59a9510df9f8\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") "
Oct 30 19:35:21.801757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801525    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-socket\") pod \"af7a4772-c323-4771-b082-59a9510df9f8\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") "
Oct 30 19:35:21.801757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801565    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-data\") pod \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") "
Oct 30 19:35:21.801757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801633    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-certs\") pod \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") "
Oct 30 19:35:21.801757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801675    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-token\") pod \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") "
Oct 30 19:35:21.801757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801729    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-credential-socket\") pod \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") "
Oct 30 19:35:21.802346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801768    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istiod-ca-cert" pod "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" (UID: "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:21.802346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801783    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-credential-socket\") pod \"af7a4772-c323-4771-b082-59a9510df9f8\" (UID: \"af7a4772-c323-4771-b082-59a9510df9f8\") "
Oct 30 19:35:21.802346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.801956    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-kube-api-access-g479j\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-kube-api-access-g479j\") pod \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") "
Oct 30 19:35:21.802346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.802056    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-data" pod "af7a4772-c323-4771-b082-59a9510df9f8" (UID: "af7a4772-c323-4771-b082-59a9510df9f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.802346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.802060    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-podinfo\") pod \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\" (UID: \"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48\") "
Oct 30 19:35:21.803104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.802835    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-socket" pod "af7a4772-c323-4771-b082-59a9510df9f8" (UID: "af7a4772-c323-4771-b082-59a9510df9f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.803104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.802844    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-data" pod "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" (UID: "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.803351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.803100    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.803351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.803124    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776wg\" (UniqueName: \"kubernetes.io/projected/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d-kube-api-access-776wg\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.803351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.803141    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.803351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.803155    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.803351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.803167    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.803634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.803358    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-socket" pod "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" (UID: "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.803634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.803531    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-credential-socket" pod "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" (UID: "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.804025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.804004    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-credential-socket" pod "af7a4772-c323-4771-b082-59a9510df9f8" (UID: "af7a4772-c323-4771-b082-59a9510df9f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.804147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.804119    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-envoy" pod "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" (UID: "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.804474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.804449    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7a4772-c323-4771-b082-59a9510df9f8-istiod-ca-cert" pod "af7a4772-c323-4771-b082-59a9510df9f8" (UID: "af7a4772-c323-4771-b082-59a9510df9f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:21.804778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.804743    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-certs" pod "af7a4772-c323-4771-b082-59a9510df9f8" (UID: "af7a4772-c323-4771-b082-59a9510df9f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.804951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.804929    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-certs" pod "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" (UID: "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.806034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.806004    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af7a4772-c323-4771-b082-59a9510df9f8-istio-podinfo" pod "af7a4772-c323-4771-b082-59a9510df9f8" (UID: "af7a4772-c323-4771-b082-59a9510df9f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:21.806034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.806023    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-kube-api-access-4md6t" pod "af7a4772-c323-4771-b082-59a9510df9f8" (UID: "af7a4772-c323-4771-b082-59a9510df9f8"). InnerVolumeSpecName "kube-api-access-4md6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:21.806832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.806805    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-envoy" pod "af7a4772-c323-4771-b082-59a9510df9f8" (UID: "af7a4772-c323-4771-b082-59a9510df9f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:21.807144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.807117    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-podinfo" pod "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" (UID: "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:21.807534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.807508    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-istio-token" pod "af7a4772-c323-4771-b082-59a9510df9f8" (UID: "af7a4772-c323-4771-b082-59a9510df9f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:21.807987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.807963    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-token" pod "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" (UID: "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:21.808093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.808000    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-kube-api-access-g479j" pod "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" (UID: "c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48"). InnerVolumeSpecName "kube-api-access-g479j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909231    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909269    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909282    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909295    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g479j\" (UniqueName: \"kubernetes.io/projected/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-kube-api-access-g479j\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909310    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909322    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af7a4772-c323-4771-b082-59a9510df9f8-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909336    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909347    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4md6t\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-kube-api-access-4md6t\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909359    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909370    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909382    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af7a4772-c323-4771-b082-59a9510df9f8-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909393    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af7a4772-c323-4771-b082-59a9510df9f8-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909405    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af7a4772-c323-4771-b082-59a9510df9f8-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:21.919712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:21.909416    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:22.016713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.015743    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"]
Oct 30 19:35:22.016713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.016038    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy" containerID="cri-o://35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10" gracePeriod=30
Oct 30 19:35:22.166488 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:22.166450    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:35:22.362720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.362602    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:22.413957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.413901    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9" event={"ID":"af7a4772-c323-4771-b082-59a9510df9f8","Type":"ContainerDied","Data":"7d89727b2c10c8aad514a72ed3518e83ee3f9aee845bbf6f9e14b47ebf7c3e71"}
Oct 30 19:35:22.414196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.413980    4925 scope.go:122] "RemoveContainer" containerID="0c2d65904d793c8509695edead5f402e63a34f4ed61f7109d43cd1cefba2a4c5"
Oct 30 19:35:22.414196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.414179    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"
Oct 30 19:35:22.422941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.422915    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"
Oct 30 19:35:22.423312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.423279    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq" event={"ID":"2d159211-5fdd-4e5f-8e50-034d2a4e3f4d","Type":"ContainerDied","Data":"fae0da4c94b4403992bca5ced1a25e14fd8739e25d0a7b3acc282686589eabe0"}
Oct 30 19:35:22.427735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.427317    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"
Oct 30 19:35:22.427735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.427340    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf" event={"ID":"c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48","Type":"ContainerDied","Data":"91b9f4039ea7bc1523827426ce9d95a3d8f56b868a54cf64f2501874c88e455c"}
Oct 30 19:35:22.434939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.434924    4925 scope.go:122] "RemoveContainer" containerID="7b204e6d5f944e5b23cf5b1019624330ba38ddde820aa36926ef9fbfb6ed2b5f"
Oct 30 19:35:22.454114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.454089    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"]
Oct 30 19:35:22.460676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.460653    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-pqcc9"]
Oct 30 19:35:22.466986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.466969    4925 scope.go:122] "RemoveContainer" containerID="a2c95115997cb98f1c0da1c64a9a2b72d0e545615fba8e3af0a14ce45dcbdcd0"
Oct 30 19:35:22.479228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.479211    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"]
Oct 30 19:35:22.486274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.486254    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d52364-8f7bc4cc6-fvcdq"]
Oct 30 19:35:22.501557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.501536    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"]
Oct 30 19:35:22.504883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:22.504863    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d52364-openshift-default-fcdf865d7-qztpf"]
Oct 30 19:35:23.264944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:23.264905    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" path="/var/lib/kubelet/pods/2d159211-5fdd-4e5f-8e50-034d2a4e3f4d/volumes"
Oct 30 19:35:23.265547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:23.265522    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af7a4772-c323-4771-b082-59a9510df9f8" path="/var/lib/kubelet/pods/af7a4772-c323-4771-b082-59a9510df9f8/volumes"
Oct 30 19:35:23.266044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:23.266028    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" path="/var/lib/kubelet/pods/c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48/volumes"
Oct 30 19:35:23.266770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:23.266753    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08e19cf-765f-46fc-997c-c25e6a7dae3a" path="/var/lib/kubelet/pods/f08e19cf-765f-46fc-997c-c25e6a7dae3a/volumes"
Oct 30 19:35:23.480778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:23.480742    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:23.661267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:23.661237    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:23.931135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:23.931040    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:24.482335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.482296    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:24.519554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.519516    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm" podUID="e2c102d1-6e7f-4f2f-b00c-c2372298d20a" containerName="main" containerID="cri-o://b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0" gracePeriod=45
Oct 30 19:35:24.678034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.677976    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:24.846733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.846705    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:35:24.877080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.877048    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-model-cache\") pod \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") "
Oct 30 19:35:24.877323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.877304    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-model-cache" pod "e2c102d1-6e7f-4f2f-b00c-c2372298d20a" (UID: "e2c102d1-6e7f-4f2f-b00c-c2372298d20a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:24.877546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.877526    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tmp-dir\") pod \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") "
Oct 30 19:35:24.877745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.877726    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-dshm\") pod \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") "
Oct 30 19:35:24.877843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.877798    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tls-certs\") pod \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") "
Oct 30 19:35:24.878239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.877921    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tmp-dir" pod "e2c102d1-6e7f-4f2f-b00c-c2372298d20a" (UID: "e2c102d1-6e7f-4f2f-b00c-c2372298d20a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:24.878239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.877942    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-kube-api-access-fgn6h\" (UniqueName: \"kubernetes.io/projected/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-kube-api-access-fgn6h\") pod \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") "
Oct 30 19:35:24.878239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.877971    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-home\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-home\") pod \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\" (UID: \"e2c102d1-6e7f-4f2f-b00c-c2372298d20a\") "
Oct 30 19:35:24.878506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.878482    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-home" pod "e2c102d1-6e7f-4f2f-b00c-c2372298d20a" (UID: "e2c102d1-6e7f-4f2f-b00c-c2372298d20a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:24.878854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.878831    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:24.879929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.878858    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:24.879929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.878871    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:24.886754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.882163    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-dshm" pod "e2c102d1-6e7f-4f2f-b00c-c2372298d20a" (UID: "e2c102d1-6e7f-4f2f-b00c-c2372298d20a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:24.886754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.882265    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tls-certs" pod "e2c102d1-6e7f-4f2f-b00c-c2372298d20a" (UID: "e2c102d1-6e7f-4f2f-b00c-c2372298d20a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:24.886754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.882783    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-kube-api-access-fgn6h" pod "e2c102d1-6e7f-4f2f-b00c-c2372298d20a" (UID: "e2c102d1-6e7f-4f2f-b00c-c2372298d20a"). InnerVolumeSpecName "kube-api-access-fgn6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:24.930671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.930644    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:24.980913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.980886    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:24.980913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.980912    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:24.981090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:24.980922    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgn6h\" (UniqueName: \"kubernetes.io/projected/e2c102d1-6e7f-4f2f-b00c-c2372298d20a-kube-api-access-fgn6h\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.277934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.277901    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"]
Oct 30 19:35:25.279144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279119    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" containerName="payload-processing"
Oct 30 19:35:25.279144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279145    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" containerName="payload-processing"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279207    4925 cpu_manager.go:414] "containerMap: removing container" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279216    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279234    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279241    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279263    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f08e19cf-765f-46fc-997c-c25e6a7dae3a" containerName="payload-processing"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279270    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08e19cf-765f-46fc-997c-c25e6a7dae3a" containerName="payload-processing"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279293    4925 cpu_manager.go:414] "containerMap: removing container" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279300    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279317    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e2c102d1-6e7f-4f2f-b00c-c2372298d20a" containerName="main"
Oct 30 19:35:25.279334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279324    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c102d1-6e7f-4f2f-b00c-c2372298d20a" containerName="main"
Oct 30 19:35:25.279921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279478    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy"
Oct 30 19:35:25.279921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279491    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fa0ab1-97e7-453d-bcd3-bac5b8f5df48" containerName="istio-proxy"
Oct 30 19:35:25.279921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279502    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08e19cf-765f-46fc-997c-c25e6a7dae3a" containerName="payload-processing"
Oct 30 19:35:25.279921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279514    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7a4772-c323-4771-b082-59a9510df9f8" containerName="istio-proxy"
Oct 30 19:35:25.279921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279525    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c102d1-6e7f-4f2f-b00c-c2372298d20a" containerName="main"
Oct 30 19:35:25.279921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.279537    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d159211-5fdd-4e5f-8e50-034d2a4e3f4d" containerName="payload-processing"
Oct 30 19:35:25.284200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.284174    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.296382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.296338    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"]
Oct 30 19:35:25.302786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.299234    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-0a6e66-openshift-default-dockercfg-9mmqk\""
Oct 30 19:35:25.305568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.305547    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:35:25.338403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.338376    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:35:25.376162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.376115    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:25.418288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.418248    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-kube-api-access-x5nvv\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-kube-api-access-x5nvv\") pod \"b03fe9ab-eebf-438a-bc73-d00f216171c6\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.609798    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/715601fa-f650-42e8-a545-ab33d4f3179c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715601fa-f650-42e8-a545-ab33d4f3179c-istio-podinfo\") pod \"715601fa-f650-42e8-a545-ab33d4f3179c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.609865    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-istio-token\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-istio-token\") pod \"715601fa-f650-42e8-a545-ab33d4f3179c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.609945    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-socket\") pod \"715601fa-f650-42e8-a545-ab33d4f3179c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610033    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03fe9ab-eebf-438a-bc73-d00f216171c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03fe9ab-eebf-438a-bc73-d00f216171c6-istiod-ca-cert\") pod \"b03fe9ab-eebf-438a-bc73-d00f216171c6\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610195    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-kube-api-access-9kl8g\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-kube-api-access-9kl8g\") pod \"715601fa-f650-42e8-a545-ab33d4f3179c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610245    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-podinfo\") pod \"b03fe9ab-eebf-438a-bc73-d00f216171c6\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610276    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-certs\") pod \"715601fa-f650-42e8-a545-ab33d4f3179c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610319    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-socket\") pod \"b03fe9ab-eebf-438a-bc73-d00f216171c6\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610418    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-certs\") pod \"b03fe9ab-eebf-438a-bc73-d00f216171c6\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610448    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-envoy\") pod \"b03fe9ab-eebf-438a-bc73-d00f216171c6\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610476    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715601fa-f650-42e8-a545-ab33d4f3179c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715601fa-f650-42e8-a545-ab33d4f3179c-istiod-ca-cert\") pod \"715601fa-f650-42e8-a545-ab33d4f3179c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610507    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-envoy\") pod \"715601fa-f650-42e8-a545-ab33d4f3179c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610574    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-data\") pod \"715601fa-f650-42e8-a545-ab33d4f3179c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610638    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-credential-socket\") pod \"715601fa-f650-42e8-a545-ab33d4f3179c\" (UID: \"715601fa-f650-42e8-a545-ab33d4f3179c\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610735    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-token\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-token\") pod \"b03fe9ab-eebf-438a-bc73-d00f216171c6\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610794    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-data\") pod \"b03fe9ab-eebf-438a-bc73-d00f216171c6\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.610838    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-credential-socket\") pod \"b03fe9ab-eebf-438a-bc73-d00f216171c6\" (UID: \"b03fe9ab-eebf-438a-bc73-d00f216171c6\") "
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.611281    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.611378    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc119c21-05da-46e6-a79b-f87ed81e7f95-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.611427    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.611505    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.611593    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.611781    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.611834    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.611981    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-kube-api-access-zbg72\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.612121    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.616765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.503462    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-kube-api-access-x5nvv" pod "b03fe9ab-eebf-438a-bc73-d00f216171c6" (UID: "b03fe9ab-eebf-438a-bc73-d00f216171c6"). InnerVolumeSpecName "kube-api-access-x5nvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:25.619233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.618900    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-data" pod "715601fa-f650-42e8-a545-ab33d4f3179c" (UID: "715601fa-f650-42e8-a545-ab33d4f3179c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.659746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.642770    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-certs" pod "715601fa-f650-42e8-a545-ab33d4f3179c" (UID: "715601fa-f650-42e8-a545-ab33d4f3179c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.659746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.643129    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03fe9ab-eebf-438a-bc73-d00f216171c6-istiod-ca-cert" pod "b03fe9ab-eebf-438a-bc73-d00f216171c6" (UID: "b03fe9ab-eebf-438a-bc73-d00f216171c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:25.659746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.650832    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-socket" pod "715601fa-f650-42e8-a545-ab33d4f3179c" (UID: "715601fa-f650-42e8-a545-ab33d4f3179c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.660814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.660764    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-certs" pod "b03fe9ab-eebf-438a-bc73-d00f216171c6" (UID: "b03fe9ab-eebf-438a-bc73-d00f216171c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.661420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.661398    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-credential-socket" pod "715601fa-f650-42e8-a545-ab33d4f3179c" (UID: "715601fa-f650-42e8-a545-ab33d4f3179c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.685865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.679076    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-credential-socket" pod "b03fe9ab-eebf-438a-bc73-d00f216171c6" (UID: "b03fe9ab-eebf-438a-bc73-d00f216171c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.686489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.685800    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-socket" pod "b03fe9ab-eebf-438a-bc73-d00f216171c6" (UID: "b03fe9ab-eebf-438a-bc73-d00f216171c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.686667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.686642    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:25.686929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.686902    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-data" pod "b03fe9ab-eebf-438a-bc73-d00f216171c6" (UID: "b03fe9ab-eebf-438a-bc73-d00f216171c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.717021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.713591    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.717021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.714026    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.722907    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715601fa-f650-42e8-a545-ab33d4f3179c-istiod-ca-cert" pod "715601fa-f650-42e8-a545-ab33d4f3179c" (UID: "715601fa-f650-42e8-a545-ab33d4f3179c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.743853    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-istio-token" pod "715601fa-f650-42e8-a545-ab33d4f3179c" (UID: "715601fa-f650-42e8-a545-ab33d4f3179c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.753166    4925 generic.go:356] "Generic (PLEG): container finished" podID="715601fa-f650-42e8-a545-ab33d4f3179c" containerID="581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383" exitCode=0
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.753409    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" event={"ID":"715601fa-f650-42e8-a545-ab33d4f3179c","Type":"ContainerDied","Data":"581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383"}
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.753463    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c" event={"ID":"715601fa-f650-42e8-a545-ab33d4f3179c","Type":"ContainerDied","Data":"e445927d93bbc29baf15f556775a11e5c1c9e9bde83dac1e04b000e1e208f9b0"}
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.753505    4925 scope.go:122] "RemoveContainer" containerID="581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.753707    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.794851    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.795000    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-kube-api-access-zbg72\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.795125    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.795341    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.795436    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc119c21-05da-46e6-a79b-f87ed81e7f95-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.795482    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.795556    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.795649    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.796189    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.796524    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.796806    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.802662    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc119c21-05da-46e6-a79b-f87ed81e7f95-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.803004    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.822911    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03fe9ab-eebf-438a-bc73-d00f216171c6-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.822952    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.822970    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.822987    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.823015    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715601fa-f650-42e8-a545-ab33d4f3179c-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.823033    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.823047    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.823096    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.823110    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.823128    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5nvv\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-kube-api-access-x5nvv\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.839719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.823143    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.875298    4925 generic.go:356] "Generic (PLEG): container finished" podID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerID="bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c" exitCode=0
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.875435    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.875574    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" event={"ID":"b03fe9ab-eebf-438a-bc73-d00f216171c6","Type":"ContainerDied","Data":"bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c"}
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.875630    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz" event={"ID":"b03fe9ab-eebf-438a-bc73-d00f216171c6","Type":"ContainerDied","Data":"e89f0b76ffacea537bb9dd9820a9fc5735a01162066170d51080d7151ca3ddbf"}
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.879056    4925 generic.go:356] "Generic (PLEG): container finished" podID="e2c102d1-6e7f-4f2f-b00c-c2372298d20a" containerID="b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0" exitCode=0
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.879194    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm" event={"ID":"e2c102d1-6e7f-4f2f-b00c-c2372298d20a","Type":"ContainerDied","Data":"b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0"}
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.879229    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm" event={"ID":"e2c102d1-6e7f-4f2f-b00c-c2372298d20a","Type":"ContainerDied","Data":"854cfdbeb6951e4e198192f3eb42aacfc067d61da1ac6059aa7aa97da5843c23"}
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.879338    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.892120    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.892312    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/715601fa-f650-42e8-a545-ab33d4f3179c-istio-podinfo" pod "715601fa-f650-42e8-a545-ab33d4f3179c" (UID: "715601fa-f650-42e8-a545-ab33d4f3179c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.892368    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-envoy" pod "b03fe9ab-eebf-438a-bc73-d00f216171c6" (UID: "b03fe9ab-eebf-438a-bc73-d00f216171c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.892400    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-kube-api-access-9kl8g" pod "715601fa-f650-42e8-a545-ab33d4f3179c" (UID: "715601fa-f650-42e8-a545-ab33d4f3179c"). InnerVolumeSpecName "kube-api-access-9kl8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.892415    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-podinfo" pod "b03fe9ab-eebf-438a-bc73-d00f216171c6" (UID: "b03fe9ab-eebf-438a-bc73-d00f216171c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.892479    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-envoy" pod "715601fa-f650-42e8-a545-ab33d4f3179c" (UID: "715601fa-f650-42e8-a545-ab33d4f3179c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.893206    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.894319    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.900943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.896098    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-kube-api-access-zbg72\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.902640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.902599    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-token" pod "b03fe9ab-eebf-438a-bc73-d00f216171c6" (UID: "b03fe9ab-eebf-438a-bc73-d00f216171c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:25.917311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.917193    4925 scope.go:122] "RemoveContainer" containerID="581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383"
Oct 30 19:35:25.917614 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:25.917589    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383\": container with ID starting with 581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383 not found: ID does not exist" containerID="581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383"
Oct 30 19:35:25.917721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.917631    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383"} err="failed to get container status \"581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383\": rpc error: code = NotFound desc = could not find container \"581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383\": container with ID starting with 581a89911d224db102b5e3d1863d10f3832865e702ccd8fee02f7d36c43d4383 not found: ID does not exist"
Oct 30 19:35:25.917721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.917655    4925 scope.go:122] "RemoveContainer" containerID="bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c"
Oct 30 19:35:25.925316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.925247    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kl8g\" (UniqueName: \"kubernetes.io/projected/715601fa-f650-42e8-a545-ab33d4f3179c-kube-api-access-9kl8g\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.925316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.925275    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.925775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.925292    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.925775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.925725    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715601fa-f650-42e8-a545-ab33d4f3179c-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.925775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.925742    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03fe9ab-eebf-438a-bc73-d00f216171c6-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.925775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.925756    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715601fa-f650-42e8-a545-ab33d4f3179c-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:25.938402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.938162    4925 scope.go:122] "RemoveContainer" containerID="bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c"
Oct 30 19:35:25.938402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.938275    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:25.950827 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:25.941976    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c\": container with ID starting with bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c not found: ID does not exist" containerID="bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c"
Oct 30 19:35:25.950827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.942015    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c"} err="failed to get container status \"bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c\": rpc error: code = NotFound desc = could not find container \"bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c\": container with ID starting with bd721141c9e9547753f666f533a259e96b96966f22b423b8689ea7adc1a47c5c not found: ID does not exist"
Oct 30 19:35:25.950827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.942039    4925 scope.go:122] "RemoveContainer" containerID="b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0"
Oct 30 19:35:25.950827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.942484    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:25.951291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.951265    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"]
Oct 30 19:35:25.957784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.957762    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/auth-test-model-618d93-kserve-689b84985c-5gdlm"]
Oct 30 19:35:25.965224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.965154    4925 scope.go:122] "RemoveContainer" containerID="b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0"
Oct 30 19:35:25.971968 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:25.971841    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0\": container with ID starting with b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0 not found: ID does not exist" containerID="b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0"
Oct 30 19:35:25.971968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:25.971909    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0"} err="failed to get container status \"b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0\": rpc error: code = NotFound desc = could not find container \"b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0\": container with ID starting with b7eebef9cd8832b393655cb66a1c197b4334782d25f00b525664d87b494d23d0 not found: ID does not exist"
Oct 30 19:35:26.084116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.084049    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"]
Oct 30 19:35:26.091060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.091029    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-56g9c"]
Oct 30 19:35:26.203350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.203316    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"]
Oct 30 19:35:26.208330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.208306    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-02feaa-openshift-default-7b4c9dd4c7-jzhwz"]
Oct 30 19:35:26.272418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.272393    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"]
Oct 30 19:35:26.273879 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:35:26.273853    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc119c21_05da_46e6_a79b_f87ed81e7f95.slice/crio-2f29bab78ef5749c9d7ea7b42a53ca3e66487a58fde9a6bbe0bad783998d6f14 WatchSource:0}: Error finding container 2f29bab78ef5749c9d7ea7b42a53ca3e66487a58fde9a6bbe0bad783998d6f14: Status 404 returned error can't find the container with id 2f29bab78ef5749c9d7ea7b42a53ca3e66487a58fde9a6bbe0bad783998d6f14
Oct 30 19:35:26.276860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.276826    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:26.276957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.276911    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:26.277029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.276956    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:35:26.663074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.663046    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:26.892488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.892454    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" event={"ID":"bc119c21-05da-46e6-a79b-f87ed81e7f95","Type":"ContainerStarted","Data":"40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7"}
Oct 30 19:35:26.892759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.892734    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" event={"ID":"bc119c21-05da-46e6-a79b-f87ed81e7f95","Type":"ContainerStarted","Data":"2f29bab78ef5749c9d7ea7b42a53ca3e66487a58fde9a6bbe0bad783998d6f14"}
Oct 30 19:35:26.896753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.896725    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:26.917258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.917202    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podStartSLOduration=1.917185065 podStartE2EDuration="1.917185065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:26.91450016 +0000 UTC m=+1761.817513196" watchObservedRunningTime="2026-10-30 19:35:26.917185065 +0000 UTC m=+1761.820198101"
Oct 30 19:35:26.939289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:26.939260    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:27.102752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.095652    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:27.345716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.344104    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:27.423324 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.422898    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" path="/var/lib/kubelet/pods/715601fa-f650-42e8-a545-ab33d4f3179c/volumes"
Oct 30 19:35:27.435840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.424005    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" path="/var/lib/kubelet/pods/b03fe9ab-eebf-438a-bc73-d00f216171c6/volumes"
Oct 30 19:35:27.435840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.424881    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c102d1-6e7f-4f2f-b00c-c2372298d20a" path="/var/lib/kubelet/pods/e2c102d1-6e7f-4f2f-b00c-c2372298d20a/volumes"
Oct 30 19:35:27.452994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.452965    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:35:27.453133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.453012    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 19:35:27.583146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.579273    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:27.670836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.670810    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:35:27.740772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.740667    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istiod-ca-cert\") pod \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") "
Oct 30 19:35:27.740956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.740829    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-credential-socket\") pod \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") "
Oct 30 19:35:27.740956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.740909    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-envoy\") pod \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") "
Oct 30 19:35:27.741082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.740996    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-socket\") pod \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") "
Oct 30 19:35:27.741144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.741115    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-podinfo\") pod \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") "
Oct 30 19:35:27.741199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.741152    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-kube-api-access-8x6rb\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-kube-api-access-8x6rb\") pod \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") "
Oct 30 19:35:27.741260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.741206    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-token\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-token\") pod \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") "
Oct 30 19:35:27.741320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.741297    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-data\") pod \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") "
Oct 30 19:35:27.741381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.741356    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-certs\") pod \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\" (UID: \"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf\") "
Oct 30 19:35:27.741971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.741127    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istiod-ca-cert" pod "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" (UID: "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:27.742108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.741969    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-socket" pod "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" (UID: "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:27.742408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.742274    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:27.742408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.742299    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:27.742821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.742793    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-data" pod "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" (UID: "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:27.742907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.742843    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-certs" pod "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" (UID: "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:27.742907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.742862    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-credential-socket" pod "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" (UID: "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:27.746990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.746956    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-token" pod "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" (UID: "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:27.749113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.749090    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-podinfo" pod "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" (UID: "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:27.749677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.749653    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:27.763578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.763548    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-envoy" pod "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" (UID: "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:27.764268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.764244    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-kube-api-access-8x6rb" pod "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" (UID: "3c002067-6cf0-430e-aeaa-f7aeb26a4cbf"). InnerVolumeSpecName "kube-api-access-8x6rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:27.844152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.844116    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:27.844152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.844148    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x6rb\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-kube-api-access-8x6rb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:27.844369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.844163    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:27.844369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.844176    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:27.844369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.844185    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:27.844369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.844194    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:27.844369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.844202    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:27.904142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.904109    4925 generic.go:356] "Generic (PLEG): container finished" podID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerID="35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10" exitCode=0
Oct 30 19:35:27.904322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.904165    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"
Oct 30 19:35:27.904322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.904202    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" event={"ID":"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf","Type":"ContainerDied","Data":"35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10"}
Oct 30 19:35:27.904322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.904259    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86" event={"ID":"3c002067-6cf0-430e-aeaa-f7aeb26a4cbf","Type":"ContainerDied","Data":"c8e43162fc78dbb60f73b8df12fe1e6328aa044726f00275e0eb87fdbdae5422"}
Oct 30 19:35:27.904322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.904286    4925 scope.go:122] "RemoveContainer" containerID="35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10"
Oct 30 19:35:27.920460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.920443    4925 scope.go:122] "RemoveContainer" containerID="35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10"
Oct 30 19:35:27.920785 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:27.920756    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10\": container with ID starting with 35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10 not found: ID does not exist" containerID="35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10"
Oct 30 19:35:27.920864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.920796    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10"} err="failed to get container status \"35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10\": rpc error: code = NotFound desc = could not find container \"35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10\": container with ID starting with 35947f1dc6b10cedb399c11cd327d24479a87d0be934a4e9948b4ece1d550d10 not found: ID does not exist"
Oct 30 19:35:27.930060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.930034    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"]
Oct 30 19:35:27.936495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.936470    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-4hs86"]
Oct 30 19:35:27.946019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:27.946000    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:28.669367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:28.669334    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:28.912582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:28.912556    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:28.913514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:28.913491    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:35:29.250535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:29.250505    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:29.265555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:29.265506    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" path="/var/lib/kubelet/pods/3c002067-6cf0-430e-aeaa-f7aeb26a4cbf/volumes"
Oct 30 19:35:29.448838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:29.448800    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:29.668203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:29.668160    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:30.459712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.459663    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"]
Oct 30 19:35:30.460407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.460383    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy"
Oct 30 19:35:30.460407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.460406    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy"
Oct 30 19:35:30.460582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.460434    4925 cpu_manager.go:414] "containerMap: removing container" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy"
Oct 30 19:35:30.460582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.460442    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy"
Oct 30 19:35:30.460582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.460474    4925 cpu_manager.go:414] "containerMap: removing container" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy"
Oct 30 19:35:30.460582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.460480    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy"
Oct 30 19:35:30.460837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.460603    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03fe9ab-eebf-438a-bc73-d00f216171c6" containerName="istio-proxy"
Oct 30 19:35:30.460837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.460617    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="715601fa-f650-42e8-a545-ab33d4f3179c" containerName="istio-proxy"
Oct 30 19:35:30.460837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.460633    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c002067-6cf0-430e-aeaa-f7aeb26a4cbf" containerName="istio-proxy"
Oct 30 19:35:30.461152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.461129    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:30.465744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.465727    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:30.471529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.471510    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-0a6e66\""
Oct 30 19:35:30.487586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.481282    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"]
Oct 30 19:35:30.487586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.481522    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-0a6e66-dockercfg-bv4wh\""
Oct 30 19:35:30.532046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.532012    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"]
Oct 30 19:35:30.536516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.536496    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:30.546503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.546480    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"]
Oct 30 19:35:30.613970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.613932    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abbd9ae-e288-40aa-a2ed-f12cf4073099-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2\" (UID: \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:30.614325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.614301    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdspd\" (UniqueName: \"kubernetes.io/projected/73962804-938e-4b9c-b058-a0f071e29de5-kube-api-access-kdspd\") pod \"payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt\" (UID: \"73962804-938e-4b9c-b058-a0f071e29de5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:30.614610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.614594    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73962804-938e-4b9c-b058-a0f071e29de5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt\" (UID: \"73962804-938e-4b9c-b058-a0f071e29de5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:30.614826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.614811    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnb6\" (UniqueName: \"kubernetes.io/projected/9abbd9ae-e288-40aa-a2ed-f12cf4073099-kube-api-access-cnnb6\") pod \"payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2\" (UID: \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:30.686853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.686821    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:30.727824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.727738    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73962804-938e-4b9c-b058-a0f071e29de5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt\" (UID: \"73962804-938e-4b9c-b058-a0f071e29de5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:30.729071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.729045    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnb6\" (UniqueName: \"kubernetes.io/projected/9abbd9ae-e288-40aa-a2ed-f12cf4073099-kube-api-access-cnnb6\") pod \"payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2\" (UID: \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:30.810726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.807680    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abbd9ae-e288-40aa-a2ed-f12cf4073099-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2\" (UID: \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:30.810726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.807803    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdspd\" (UniqueName: \"kubernetes.io/projected/73962804-938e-4b9c-b058-a0f071e29de5-kube-api-access-kdspd\") pod \"payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt\" (UID: \"73962804-938e-4b9c-b058-a0f071e29de5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:30.810726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.728892    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73962804-938e-4b9c-b058-a0f071e29de5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt\" (UID: \"73962804-938e-4b9c-b058-a0f071e29de5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:30.810726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.809321    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abbd9ae-e288-40aa-a2ed-f12cf4073099-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2\" (UID: \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:30.851720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.833755    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnb6\" (UniqueName: \"kubernetes.io/projected/9abbd9ae-e288-40aa-a2ed-f12cf4073099-kube-api-access-cnnb6\") pod \"payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2\" (UID: \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:30.868918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.859244    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:30.892018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.890776    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:30.929663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:30.929352    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdspd\" (UniqueName: \"kubernetes.io/projected/73962804-938e-4b9c-b058-a0f071e29de5-kube-api-access-kdspd\") pod \"payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt\" (UID: \"73962804-938e-4b9c-b058-a0f071e29de5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:31.085051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.085013    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:31.246073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.246028    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"]
Oct 30 19:35:31.250785 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:35:31.250731    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abbd9ae_e288_40aa_a2ed_f12cf4073099.slice/crio-5e28802889976b4f80c7bc24c293ef7fbad9a4b66fa2b71e86b482f90fac64b6 WatchSource:0}: Error finding container 5e28802889976b4f80c7bc24c293ef7fbad9a4b66fa2b71e86b482f90fac64b6: Status 404 returned error can't find the container with id 5e28802889976b4f80c7bc24c293ef7fbad9a4b66fa2b71e86b482f90fac64b6
Oct 30 19:35:31.396295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.394441    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"]
Oct 30 19:35:31.396295 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:35:31.395224    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73962804_938e_4b9c_b058_a0f071e29de5.slice/crio-680e98d07d008d1675dae291a3773427dc90fca742e29a7369183d1f341fbe45 WatchSource:0}: Error finding container 680e98d07d008d1675dae291a3773427dc90fca742e29a7369183d1f341fbe45: Status 404 returned error can't find the container with id 680e98d07d008d1675dae291a3773427dc90fca742e29a7369183d1f341fbe45
Oct 30 19:35:31.414257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.414215    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:31.627766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.626526    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:31.939370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.939335    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2" event={"ID":"9abbd9ae-e288-40aa-a2ed-f12cf4073099","Type":"ContainerStarted","Data":"984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53"}
Oct 30 19:35:31.939594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.939388    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2" event={"ID":"9abbd9ae-e288-40aa-a2ed-f12cf4073099","Type":"ContainerStarted","Data":"5e28802889976b4f80c7bc24c293ef7fbad9a4b66fa2b71e86b482f90fac64b6"}
Oct 30 19:35:31.939594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.939573    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:31.941488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.941462    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt" event={"ID":"73962804-938e-4b9c-b058-a0f071e29de5","Type":"ContainerStarted","Data":"bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e"}
Oct 30 19:35:31.941600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.941502    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt" event={"ID":"73962804-938e-4b9c-b058-a0f071e29de5","Type":"ContainerStarted","Data":"680e98d07d008d1675dae291a3773427dc90fca742e29a7369183d1f341fbe45"}
Oct 30 19:35:31.941600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.941537    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:31.964888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:31.964837    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2" podStartSLOduration=1.964823231 podStartE2EDuration="1.964823231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:31.964318543 +0000 UTC m=+1766.867331584" watchObservedRunningTime="2026-10-30 19:35:31.964823231 +0000 UTC m=+1766.867836269"
Oct 30 19:35:32.086848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:32.086815    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:32.752323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:32.752292    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:33.348766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.348124    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt" podStartSLOduration=3.348102826 podStartE2EDuration="3.348102826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:31.99437239 +0000 UTC m=+1766.897385429" watchObservedRunningTime="2026-10-30 19:35:33.348102826 +0000 UTC m=+1768.251115864"
Oct 30 19:35:33.401439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.389245    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"]
Oct 30 19:35:33.425493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.419981    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"]
Oct 30 19:35:33.425493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.420100    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.429127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.429102    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-618d93\"/\"default-dockercfg-9nqgt\""
Oct 30 19:35:33.442339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.438205    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-618d93\"/\"kube-root-ca.crt\""
Oct 30 19:35:33.442339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.439160    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-618d93\"/\"openshift-service-ca.crt\""
Oct 30 19:35:33.442339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.439373    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-618d93\"/\"rate-test-model-618d93-kserve-self-signed-certs\""
Oct 30 19:35:33.474722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.468583    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:33.572711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.564140    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-dshm\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.572711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.564255    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/399ffc7e-58f7-4662-9745-c2feddef06cb-tls-certs\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.572711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.567804    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-tmp-dir\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.579711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.574019    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-home\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.579711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.574092    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnzmf\" (UniqueName: \"kubernetes.io/projected/399ffc7e-58f7-4662-9745-c2feddef06cb-kube-api-access-gnzmf\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.579711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.574179    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-model-cache\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.717713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.711133    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/399ffc7e-58f7-4662-9745-c2feddef06cb-tls-certs\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.717713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.711220    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-tmp-dir\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.717713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.711417    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-home\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.717713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.711456    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzmf\" (UniqueName: \"kubernetes.io/projected/399ffc7e-58f7-4662-9745-c2feddef06cb-kube-api-access-gnzmf\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.717713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.711517    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-model-cache\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.717713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.711653    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-dshm\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.743846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.743808    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-home\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.769710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.744401    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-model-cache\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.769710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.754988    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-tmp-dir\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.772063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.772026    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-dshm\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.803715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.775661    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzmf\" (UniqueName: \"kubernetes.io/projected/399ffc7e-58f7-4662-9745-c2feddef06cb-kube-api-access-gnzmf\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.812712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.809822    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/399ffc7e-58f7-4662-9745-c2feddef06cb-tls-certs\") pod \"rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") " pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:33.888713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:33.881172    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:34.065698 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:34.065593    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:34.214247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:34.213880    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:34.420164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:34.420139    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"]
Oct 30 19:35:34.422513 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:35:34.422481    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod399ffc7e_58f7_4662_9745_c2feddef06cb.slice/crio-0b657e6c63cafaf820716c4ca8c7e2dc9389b2c2de541d0c0d512440e7b1e425 WatchSource:0}: Error finding container 0b657e6c63cafaf820716c4ca8c7e2dc9389b2c2de541d0c0d512440e7b1e425: Status 404 returned error can't find the container with id 0b657e6c63cafaf820716c4ca8c7e2dc9389b2c2de541d0c0d512440e7b1e425
Oct 30 19:35:34.515944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:34.513522    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:34.676284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:34.676179    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:34.856801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:34.856240    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:35.001721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:35.001555    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" event={"ID":"399ffc7e-58f7-4662-9745-c2feddef06cb","Type":"ContainerStarted","Data":"ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6"}
Oct 30 19:35:35.001721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:35.001617    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" event={"ID":"399ffc7e-58f7-4662-9745-c2feddef06cb","Type":"ContainerStarted","Data":"0b657e6c63cafaf820716c4ca8c7e2dc9389b2c2de541d0c0d512440e7b1e425"}
Oct 30 19:35:35.038482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:35.027518    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podStartSLOduration=2.027499259 podStartE2EDuration="2.027499259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:35.025075285 +0000 UTC m=+1769.928088324" watchObservedRunningTime="2026-10-30 19:35:35.027499259 +0000 UTC m=+1769.930512299"
Oct 30 19:35:35.059020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:35.058987    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:35.066756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:35.066720    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:35.278750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:35.278194    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:35.714580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:35.714549    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:36.077512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:36.077352    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:36.809575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:36.809533    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:37.050167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:37.050067    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:38.088734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:38.088699    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:38.804830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:38.804795    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:39.508093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:39.508064    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:40.499666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:40.499615    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:41.242521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:41.242486    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:41.842285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:41.842249    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:42.156285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.156255    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"]
Oct 30 19:35:42.164965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.164938    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.168590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.168562    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"]
Oct 30 19:35:42.172945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.172872    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1fa500\"/\"openshift-service-ca.crt\""
Oct 30 19:35:42.173094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.173072    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1fa500\"/\"test-model-1fa500-kserve-self-signed-certs\""
Oct 30 19:35:42.173168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.173096    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1fa500\"/\"default-dockercfg-n5xlm\""
Oct 30 19:35:42.173168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.173164    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1fa500\"/\"kube-root-ca.crt\""
Oct 30 19:35:42.202898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.202869    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-dshm\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.203087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.203067    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tls-certs\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.203136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.203123    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-home\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.203261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.203235    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-model-cache\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.203433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.203416    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tmp-dir\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.203468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.203455    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvxsw\" (UniqueName: \"kubernetes.io/projected/a62f41cc-7f6d-4be3-b26d-24c474350cb5-kube-api-access-xvxsw\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.261613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.261578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:42.304888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.304848    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tmp-dir\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.305060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.304903    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvxsw\" (UniqueName: \"kubernetes.io/projected/a62f41cc-7f6d-4be3-b26d-24c474350cb5-kube-api-access-xvxsw\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.305060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.305038    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-dshm\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.305269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.305242    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tls-certs\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.305373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.305282    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tmp-dir\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.305373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.305309    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-home\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.305373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.305363    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-model-cache\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.305666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.305634    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-home\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.305807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.305741    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-model-cache\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.307789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.307769    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-dshm\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.308002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.307981    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tls-certs\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.317882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.317860    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvxsw\" (UniqueName: \"kubernetes.io/projected/a62f41cc-7f6d-4be3-b26d-24c474350cb5-kube-api-access-xvxsw\") pod \"test-model-1fa500-kserve-8f96478d5-q7mz4\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") " pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.509316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.508982    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:42.900779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.900747    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:42.956533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.956508    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:35:42.957524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.957501    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:35:42.994316 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:35:42.994285    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda62f41cc_7f6d_4be3_b26d_24c474350cb5.slice/crio-8a1746fd1ad8bc48ab5a8a5357e7b08d676899a184101ba92a4c0fa9522d59d9 WatchSource:0}: Error finding container 8a1746fd1ad8bc48ab5a8a5357e7b08d676899a184101ba92a4c0fa9522d59d9: Status 404 returned error can't find the container with id 8a1746fd1ad8bc48ab5a8a5357e7b08d676899a184101ba92a4c0fa9522d59d9
Oct 30 19:35:42.998149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:42.998127    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"]
Oct 30 19:35:43.181723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:43.160891    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" event={"ID":"a62f41cc-7f6d-4be3-b26d-24c474350cb5","Type":"ContainerStarted","Data":"5636b13173e78216dc381b680fb363bb1bc72345c1b4465bdb10d8c267998dce"}
Oct 30 19:35:43.181723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:43.160954    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" event={"ID":"a62f41cc-7f6d-4be3-b26d-24c474350cb5","Type":"ContainerStarted","Data":"8a1746fd1ad8bc48ab5a8a5357e7b08d676899a184101ba92a4c0fa9522d59d9"}
Oct 30 19:35:43.225712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:43.224948    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" podStartSLOduration=1.224924069 podStartE2EDuration="1.224924069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:43.206846278 +0000 UTC m=+1778.109859319" watchObservedRunningTime="2026-10-30 19:35:43.224924069 +0000 UTC m=+1778.127937109"
Oct 30 19:35:43.514745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:43.511249    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:44.066260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:44.066219    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:44.089650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:44.085568    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:44.180730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:44.180681    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:35:44.703799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:44.698211    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:44.892343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:44.892312    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:45.417308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:45.417271    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:46.072127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:46.072095    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:46.296567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:46.296534    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:46.974162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:46.974115    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:47.247946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:47.244537    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:47.418405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:47.418337    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:47.771441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:47.771316    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:48.099123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:48.099036    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:48.549202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:48.549173    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:48.721661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:48.721599    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:48.896529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:48.896495    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:49.080047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:49.080014    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:49.284666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:49.284578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:49.627766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:49.619376    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:50.047588 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.047555    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:50.304251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.304150    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:50.562294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.562150    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:50.618705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.618664    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"]
Oct 30 19:35:50.623524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.623497    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:35:50.627457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.627432    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-8e6762\""
Oct 30 19:35:50.645989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.645967    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"]
Oct 30 19:35:50.729100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.729067    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-8e6762-dockercfg-rq5f2\""
Oct 30 19:35:50.773409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.773375    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:50.838770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.816351    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8\" (UID: \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:35:50.838770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.816455    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzt2\" (UniqueName: \"kubernetes.io/projected/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-kube-api-access-mhzt2\") pod \"payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8\" (UID: \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:35:50.938618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.938017    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8\" (UID: \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:35:50.938618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.938131    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzt2\" (UniqueName: \"kubernetes.io/projected/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-kube-api-access-mhzt2\") pod \"payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8\" (UID: \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:35:50.939587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.939558    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8\" (UID: \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:35:50.961449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:50.961419    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzt2\" (UniqueName: \"kubernetes.io/projected/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-kube-api-access-mhzt2\") pod \"payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8\" (UID: \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:35:51.249011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:51.248972    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:35:51.435264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:51.386320    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:51.551188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:51.551086    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"]
Oct 30 19:35:51.551781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:51.551749    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy" containerID="cri-o://7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224" gracePeriod=30
Oct 30 19:35:51.559471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:51.559447    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"]
Oct 30 19:35:51.562307 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:35:51.562275    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb02c995_3f69_4ced_b63f_f461c5c5aaf2.slice/crio-bf4e8be510025cbec27030a0d1336e03f02b6457b05bdf6bd21fba28bdf7d0f9 WatchSource:0}: Error finding container bf4e8be510025cbec27030a0d1336e03f02b6457b05bdf6bd21fba28bdf7d0f9: Status 404 returned error can't find the container with id bf4e8be510025cbec27030a0d1336e03f02b6457b05bdf6bd21fba28bdf7d0f9
Oct 30 19:35:52.129196 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:52.129135    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:35:52.180723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:52.173034    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:52.397728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:52.397644    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8" event={"ID":"fb02c995-3f69-4ced-b63f-f461c5c5aaf2","Type":"ContainerStarted","Data":"7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7"}
Oct 30 19:35:52.397999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:52.397977    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8" event={"ID":"fb02c995-3f69-4ced-b63f-f461c5c5aaf2","Type":"ContainerStarted","Data":"bf4e8be510025cbec27030a0d1336e03f02b6457b05bdf6bd21fba28bdf7d0f9"}
Oct 30 19:35:52.398146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:52.398132    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:35:52.417773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:52.417731    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8" podStartSLOduration=2.417718258 podStartE2EDuration="2.417718258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:52.414730477 +0000 UTC m=+1787.317743515" watchObservedRunningTime="2026-10-30 19:35:52.417718258 +0000 UTC m=+1787.320731295"
Oct 30 19:35:52.509646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:52.509615    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:52.528853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:52.528828    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:53.198422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:53.198381    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:53.417409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:53.417382    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:35:53.946108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:53.946071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:54.116577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.116546    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:54.325823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.323235    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:54.597409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.597324    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"]
Oct 30 19:35:54.603423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.603202    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.612872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.612832    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e3586f\"/\"kube-root-ca.crt\""
Oct 30 19:35:54.613029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.612921    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e3586f\"/\"openshift-service-ca.crt\""
Oct 30 19:35:54.613309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.613278    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e3586f\"/\"default-dockercfg-bpqwz\""
Oct 30 19:35:54.613433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.613315    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e3586f\"/\"rate-test-model-e3586f-kserve-self-signed-certs\""
Oct 30 19:35:54.620525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.620503    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"]
Oct 30 19:35:54.652977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.652948    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-dshm\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.653141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.653121    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlvjk\" (UniqueName: \"kubernetes.io/projected/9fb62679-e649-4990-93d7-745ec5d803b3-kube-api-access-qlvjk\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.653268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.653250    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-model-cache\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.653334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.653305    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-home\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.653396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.653361    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-tmp-dir\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.653486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.653472    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb62679-e649-4990-93d7-745ec5d803b3-tls-certs\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.711167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.711131    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:54.754440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.754407    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlvjk\" (UniqueName: \"kubernetes.io/projected/9fb62679-e649-4990-93d7-745ec5d803b3-kube-api-access-qlvjk\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.754633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.754508    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-model-cache\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.754633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.754529    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-home\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.754633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.754561    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-tmp-dir\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.754861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.754647    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb62679-e649-4990-93d7-745ec5d803b3-tls-certs\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.754861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.754755    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-dshm\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.755203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.755174    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-home\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.755269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.755220    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-model-cache\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.755471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.755439    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-tmp-dir\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.758387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.758367    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-dshm\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.758764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.758738    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb62679-e649-4990-93d7-745ec5d803b3-tls-certs\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.766857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.766806    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlvjk\" (UniqueName: \"kubernetes.io/projected/9fb62679-e649-4990-93d7-745ec5d803b3-kube-api-access-qlvjk\") pod \"rate-test-model-e3586f-kserve-85c859cc7-zj9fp\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") " pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:54.888155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.887746    4925 patch_prober.go:29] interesting pod/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:35:54.888155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.887801    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:35:54.932739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:54.927803    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:55.273719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:55.268730    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:55.311891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:55.311850    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"]
Oct 30 19:35:55.457346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:55.457304    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" event={"ID":"9fb62679-e649-4990-93d7-745ec5d803b3","Type":"ContainerStarted","Data":"8c62a7f7dc5cee89df964661d1379e07ca555ae97283a733ac1a002c4d6025f9"}
Oct 30 19:35:55.457766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:55.457364    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" event={"ID":"9fb62679-e649-4990-93d7-745ec5d803b3","Type":"ContainerStarted","Data":"bfef3933995f922d194e28e35c08ea6a4132c6b84da2913f08b667a579455bb1"}
Oct 30 19:35:55.501938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:55.501796    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:55.699981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:55.695610    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:55.940945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:55.940913    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:35:56.024105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:56.024007    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:56.216719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:56.216583    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:56.307234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:56.307114    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podStartSLOduration=2.307094154 podStartE2EDuration="2.307094154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:55.479921732 +0000 UTC m=+1790.382934770" watchObservedRunningTime="2026-10-30 19:35:56.307094154 +0000 UTC m=+1791.210107193"
Oct 30 19:35:56.307420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:56.307328    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"]
Oct 30 19:35:56.307594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:56.307570    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy" containerID="cri-o://5b17408889c7c14dd444c9cb628b28ed8c7135f5d55b223030fa23876318446f" gracePeriod=30
Oct 30 19:35:56.409340 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:56.409298    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:56.608723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:56.602634    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:56.945041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:56.945016    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:35:57.075834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.075801    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-istio-token\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-istio-token\") pod \"25918327-0d30-438b-979c-4531f0fd270f\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") "
Oct 30 19:35:57.076043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.075896    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-data\") pod \"25918327-0d30-438b-979c-4531f0fd270f\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") "
Oct 30 19:35:57.076043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.076002    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-envoy\") pod \"25918327-0d30-438b-979c-4531f0fd270f\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") "
Oct 30 19:35:57.076277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.076247    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-data" pod "25918327-0d30-438b-979c-4531f0fd270f" (UID: "25918327-0d30-438b-979c-4531f0fd270f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:57.076480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.076462    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25918327-0d30-438b-979c-4531f0fd270f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25918327-0d30-438b-979c-4531f0fd270f-istiod-ca-cert\") pod \"25918327-0d30-438b-979c-4531f0fd270f\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") "
Oct 30 19:35:57.077098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.077073    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25918327-0d30-438b-979c-4531f0fd270f-istiod-ca-cert" pod "25918327-0d30-438b-979c-4531f0fd270f" (UID: "25918327-0d30-438b-979c-4531f0fd270f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:57.077206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.077183    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-credential-socket\") pod \"25918327-0d30-438b-979c-4531f0fd270f\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") "
Oct 30 19:35:57.077321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.077306    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-certs\") pod \"25918327-0d30-438b-979c-4531f0fd270f\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") "
Oct 30 19:35:57.077434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.077419    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-kube-api-access-9q82z\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-kube-api-access-9q82z\") pod \"25918327-0d30-438b-979c-4531f0fd270f\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") "
Oct 30 19:35:57.077495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.077482    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25918327-0d30-438b-979c-4531f0fd270f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25918327-0d30-438b-979c-4531f0fd270f-istio-podinfo\") pod \"25918327-0d30-438b-979c-4531f0fd270f\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") "
Oct 30 19:35:57.077541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.077529    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-socket\") pod \"25918327-0d30-438b-979c-4531f0fd270f\" (UID: \"25918327-0d30-438b-979c-4531f0fd270f\") "
Oct 30 19:35:57.078142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.078118    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-certs" pod "25918327-0d30-438b-979c-4531f0fd270f" (UID: "25918327-0d30-438b-979c-4531f0fd270f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:57.078610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.078391    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-socket" pod "25918327-0d30-438b-979c-4531f0fd270f" (UID: "25918327-0d30-438b-979c-4531f0fd270f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:57.078776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.078757    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.078857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.078842    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.078917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.078865    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.078917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.078883    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25918327-0d30-438b-979c-4531f0fd270f-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.079373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.079147    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-credential-socket" pod "25918327-0d30-438b-979c-4531f0fd270f" (UID: "25918327-0d30-438b-979c-4531f0fd270f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:57.080464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.080364    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-envoy" pod "25918327-0d30-438b-979c-4531f0fd270f" (UID: "25918327-0d30-438b-979c-4531f0fd270f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:57.080464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.080427    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-istio-token" pod "25918327-0d30-438b-979c-4531f0fd270f" (UID: "25918327-0d30-438b-979c-4531f0fd270f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:57.081031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.081004    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25918327-0d30-438b-979c-4531f0fd270f-istio-podinfo" pod "25918327-0d30-438b-979c-4531f0fd270f" (UID: "25918327-0d30-438b-979c-4531f0fd270f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:35:57.081609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.081582    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-kube-api-access-9q82z" pod "25918327-0d30-438b-979c-4531f0fd270f" (UID: "25918327-0d30-438b-979c-4531f0fd270f"). InnerVolumeSpecName "kube-api-access-9q82z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:57.180650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.180569    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.180650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.180608    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q82z\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-kube-api-access-9q82z\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.180650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.180626    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25918327-0d30-438b-979c-4531f0fd270f-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.180650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.180639    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25918327-0d30-438b-979c-4531f0fd270f-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.180650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.180648    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25918327-0d30-438b-979c-4531f0fd270f-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:57.502291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.502199    4925 generic.go:356] "Generic (PLEG): container finished" podID="25918327-0d30-438b-979c-4531f0fd270f" containerID="7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224" exitCode=0
Oct 30 19:35:57.502291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.502255    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"
Oct 30 19:35:57.502291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.502276    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" event={"ID":"25918327-0d30-438b-979c-4531f0fd270f","Type":"ContainerDied","Data":"7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224"}
Oct 30 19:35:57.502582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.502344    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l" event={"ID":"25918327-0d30-438b-979c-4531f0fd270f","Type":"ContainerDied","Data":"95a4da07604aaa7b0f087970763258055ad443334f9dd89c748cafe4632818da"}
Oct 30 19:35:57.502582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.502375    4925 scope.go:122] "RemoveContainer" containerID="7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224"
Oct 30 19:35:57.514410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.514394    4925 scope.go:122] "RemoveContainer" containerID="a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5"
Oct 30 19:35:57.523186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.523164    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"]
Oct 30 19:35:57.524757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.524742    4925 scope.go:122] "RemoveContainer" containerID="7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224"
Oct 30 19:35:57.525018 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:57.525000    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224\": container with ID starting with 7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224 not found: ID does not exist" containerID="7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224"
Oct 30 19:35:57.525099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.525033    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224"} err="failed to get container status \"7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224\": rpc error: code = NotFound desc = could not find container \"7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224\": container with ID starting with 7e56bf218bfffeb3648909084f13085b3466c88f94b0989a09b3b00a28b9a224 not found: ID does not exist"
Oct 30 19:35:57.525099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.525059    4925 scope.go:122] "RemoveContainer" containerID="a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5"
Oct 30 19:35:57.525294 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:35:57.525278    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5\": container with ID starting with a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5 not found: ID does not exist" containerID="a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5"
Oct 30 19:35:57.525344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.525304    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5"} err="failed to get container status \"a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5\": rpc error: code = NotFound desc = could not find container \"a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5\": container with ID starting with a91ec4b44fb1587c7ced3a3a0e51299986a1a6b980c9206216613a6a9d5ce8f5 not found: ID does not exist"
Oct 30 19:35:57.533202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.533180    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-lb25l"]
Oct 30 19:35:57.612948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.612919    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:57.773877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:57.773791    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:58.531439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:58.531407    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:59.167759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:59.163612    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:59.264872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:59.264842    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25918327-0d30-438b-979c-4531f0fd270f" path="/var/lib/kubelet/pods/25918327-0d30-438b-979c-4531f0fd270f/volumes"
Oct 30 19:35:59.958398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:35:59.958361    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:00.645605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:00.645562    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:01.243943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.243903    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:01.546413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.546325    4925 generic.go:356] "Generic (PLEG): container finished" podID="587b6511-ae1d-469e-8ecd-89aede370111" containerID="5b17408889c7c14dd444c9cb628b28ed8c7135f5d55b223030fa23876318446f" exitCode=0
Oct 30 19:36:01.546413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.546368    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" event={"ID":"587b6511-ae1d-469e-8ecd-89aede370111","Type":"ContainerDied","Data":"5b17408889c7c14dd444c9cb628b28ed8c7135f5d55b223030fa23876318446f"}
Oct 30 19:36:01.608099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.608074    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:36:01.772945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.772912    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/587b6511-ae1d-469e-8ecd-89aede370111-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587b6511-ae1d-469e-8ecd-89aede370111-istio-podinfo\") pod \"587b6511-ae1d-469e-8ecd-89aede370111\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") "
Oct 30 19:36:01.773446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.772955    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-kube-api-access-2kv24\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-kube-api-access-2kv24\") pod \"587b6511-ae1d-469e-8ecd-89aede370111\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") "
Oct 30 19:36:01.773446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773065    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-istio-token\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-istio-token\") pod \"587b6511-ae1d-469e-8ecd-89aede370111\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") "
Oct 30 19:36:01.773446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773147    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-socket\") pod \"587b6511-ae1d-469e-8ecd-89aede370111\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") "
Oct 30 19:36:01.773446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773189    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-envoy\") pod \"587b6511-ae1d-469e-8ecd-89aede370111\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") "
Oct 30 19:36:01.773446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773268    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-credential-socket\") pod \"587b6511-ae1d-469e-8ecd-89aede370111\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") "
Oct 30 19:36:01.773446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773350    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-data\") pod \"587b6511-ae1d-469e-8ecd-89aede370111\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") "
Oct 30 19:36:01.773446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773382    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587b6511-ae1d-469e-8ecd-89aede370111-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587b6511-ae1d-469e-8ecd-89aede370111-istiod-ca-cert\") pod \"587b6511-ae1d-469e-8ecd-89aede370111\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") "
Oct 30 19:36:01.774002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773449    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-socket" pod "587b6511-ae1d-469e-8ecd-89aede370111" (UID: "587b6511-ae1d-469e-8ecd-89aede370111"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.774002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773464    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-certs\") pod \"587b6511-ae1d-469e-8ecd-89aede370111\" (UID: \"587b6511-ae1d-469e-8ecd-89aede370111\") "
Oct 30 19:36:01.774002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773870    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-credential-socket" pod "587b6511-ae1d-469e-8ecd-89aede370111" (UID: "587b6511-ae1d-469e-8ecd-89aede370111"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.774002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.773881    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-data" pod "587b6511-ae1d-469e-8ecd-89aede370111" (UID: "587b6511-ae1d-469e-8ecd-89aede370111"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.774233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.774087    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-certs" pod "587b6511-ae1d-469e-8ecd-89aede370111" (UID: "587b6511-ae1d-469e-8ecd-89aede370111"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.774305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.774246    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587b6511-ae1d-469e-8ecd-89aede370111-istiod-ca-cert" pod "587b6511-ae1d-469e-8ecd-89aede370111" (UID: "587b6511-ae1d-469e-8ecd-89aede370111"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:01.774611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.774588    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.774611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.774613    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.774815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.774623    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.774815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.774632    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587b6511-ae1d-469e-8ecd-89aede370111-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.774815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.774640    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.775805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.775781    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-kube-api-access-2kv24" pod "587b6511-ae1d-469e-8ecd-89aede370111" (UID: "587b6511-ae1d-469e-8ecd-89aede370111"). InnerVolumeSpecName "kube-api-access-2kv24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:01.776026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.776003    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/587b6511-ae1d-469e-8ecd-89aede370111-istio-podinfo" pod "587b6511-ae1d-469e-8ecd-89aede370111" (UID: "587b6511-ae1d-469e-8ecd-89aede370111"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:01.776093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.776026    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-istio-token" pod "587b6511-ae1d-469e-8ecd-89aede370111" (UID: "587b6511-ae1d-469e-8ecd-89aede370111"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:01.776151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.776117    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-envoy" pod "587b6511-ae1d-469e-8ecd-89aede370111" (UID: "587b6511-ae1d-469e-8ecd-89aede370111"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:01.876516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.876432    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587b6511-ae1d-469e-8ecd-89aede370111-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.876516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.876469    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kv24\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-kube-api-access-2kv24\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.876516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.876484    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587b6511-ae1d-469e-8ecd-89aede370111-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:01.876516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:01.876503    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587b6511-ae1d-469e-8ecd-89aede370111-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.126231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:02.126177    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:02.554979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:02.554945    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"
Oct 30 19:36:02.554979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:02.554966    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d" event={"ID":"587b6511-ae1d-469e-8ecd-89aede370111","Type":"ContainerDied","Data":"5b30e9ab3a226c1a79e2ce6a84e952b4759bd59aaddc8120aaa2a5523d144572"}
Oct 30 19:36:02.555254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:02.555034    4925 scope.go:122] "RemoveContainer" containerID="5b17408889c7c14dd444c9cb628b28ed8c7135f5d55b223030fa23876318446f"
Oct 30 19:36:02.617947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:02.617917    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"]
Oct 30 19:36:02.626339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:02.626302    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-69f3a9-openshift-default-64fdcd667c-cdb5d"]
Oct 30 19:36:02.881893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:02.881789    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:03.265618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:03.265583    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587b6511-ae1d-469e-8ecd-89aede370111" path="/var/lib/kubelet/pods/587b6511-ae1d-469e-8ecd-89aede370111/volumes"
Oct 30 19:36:03.409677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:03.409651    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:36:03.782837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:03.782796    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:04.352717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:04.352668    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:04.941018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:04.940983    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:36:04.953551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:04.953527    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:36:05.235784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:05.235617    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:05.592681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:05.592606    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:36:06.057712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:06.047362    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:06.267584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:06.267545    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:36:06.290995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:06.290972    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:36:06.685803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:06.681809    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:06.851565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:06.851531    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:07.652709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:07.652638    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:08.655795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:08.654217    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:09.740969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:09.740927    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:09.919375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:09.919340    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:10.134770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:10.129197    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:10.664487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:10.664454    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:11.747280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:11.747239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:12.356168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:12.355430    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:12.516408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:12.516374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:13.269715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:13.269493    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:14.147452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:14.147413    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:14.876133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:14.876100    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:15.655054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:15.655011    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:16.379463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:16.379426    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:17.121715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:17.121664    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:18.174343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:18.174301    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:18.239851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:18.239798    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f" podUID="ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.164:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:18.316559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:18.316522    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:18.460768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:18.446848    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:18.594720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:18.593941    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:36:18.594720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:18.594337    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:18.636014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:18.635869    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:18.738583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:18.738478    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:19.054634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.054544    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:19.178559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.178503    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:19.179718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.179675    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:36:19.183544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.183518    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:36:19.234828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.234790    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:19.390162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.384038    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f" podUID="ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.164:8000/ready\": context deadline exceeded"
Oct 30 19:36:19.420003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.419959    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:19.437754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.437720    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:36:19.438055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.438038    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:36:19.449548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.449500    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:19.450373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.450348    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:36:19.500857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.500580    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:36:19.768770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.741556    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:19.834290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.834159    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:36:19.852245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.851134    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:36:19.981523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:19.981492    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:20.742791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:20.742752    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:21.464958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:21.463071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:22.191666 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:36:22.191633    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:36:22.428359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:22.428324    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:23.107608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:23.107578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:23.868663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:23.868621    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:24.692201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:24.692162    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:25.393004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:25.392966    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:26.102027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:26.101941    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:27.074640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:27.074549    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:27.932586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:27.932537    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:28.914390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:28.914313    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:36:28.914858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:28.914382    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 19:36:29.024975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.024878    4925 generic.go:356] "Generic (PLEG): container finished" podID="5d846076-2d4e-42fe-8909-b6fde688f282" containerID="b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3" exitCode=0
Oct 30 19:36:29.025208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.024968    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" event={"ID":"5d846076-2d4e-42fe-8909-b6fde688f282","Type":"ContainerDied","Data":"b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3"}
Oct 30 19:36:29.025653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.025630    4925 scope.go:122] "RemoveContainer" containerID="b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3"
Oct 30 19:36:29.027784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.027762    4925 generic.go:356] "Generic (PLEG): container finished" podID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerID="40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7" exitCode=0
Oct 30 19:36:29.027896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.027795    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" event={"ID":"bc119c21-05da-46e6-a79b-f87ed81e7f95","Type":"ContainerDied","Data":"40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7"}
Oct 30 19:36:29.028180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.028148    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:29.028240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.028227    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:29.028281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.028170    4925 scope.go:122] "RemoveContainer" containerID="40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7"
Oct 30 19:36:29.028281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.028269    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:29.030142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.030111    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:29.030270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.030185    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:29.030270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.030221    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:29.508350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.508309    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:29.669885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:29.669850    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:30.074746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.074455    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" event={"ID":"5d846076-2d4e-42fe-8909-b6fde688f282","Type":"ContainerStarted","Data":"3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621"}
Oct 30 19:36:30.088897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.080307    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" event={"ID":"bc119c21-05da-46e6-a79b-f87ed81e7f95","Type":"ContainerStarted","Data":"a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397"}
Oct 30 19:36:30.161808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.161727    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:30.655586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.655544    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:30.880354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.880318    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:30.945773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.942113    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:36:30.945773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.942250    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:36:30.945773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.942297    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 19:36:30.972326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.968030    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:36:30.980714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.975803    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:36:30.980714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:30.975870    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:36:31.073819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:31.073454    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:31.941129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:31.941092    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:36:31.941775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:31.941742    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 19:36:31.966631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:31.966593    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:36:31.966830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:31.966635    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:36:32.175701 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.175656    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"]
Oct 30 19:36:32.177095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.177067    4925 cpu_manager.go:414] "containerMap: removing container" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy"
Oct 30 19:36:32.177095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.177094    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy"
Oct 30 19:36:32.177290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.177130    4925 cpu_manager.go:414] "containerMap: removing container" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy"
Oct 30 19:36:32.177290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.177137    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy"
Oct 30 19:36:32.177415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.177360    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy"
Oct 30 19:36:32.177415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.177383    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy"
Oct 30 19:36:32.177415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.177407    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="587b6511-ae1d-469e-8ecd-89aede370111" containerName="istio-proxy"
Oct 30 19:36:32.180549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.180523    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.252351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.238991    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"]
Oct 30 19:36:32.326710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.326655    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:32.466801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.445110    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.476009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.475968    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.476349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.476327    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a170-f2b9-4920-8939-5eb2e687e4df-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.476515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.476499    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.476798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.476778    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.476943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.476928    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg49m\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-kube-api-access-jg49m\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.477046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.477033    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.477206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.477191    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.477394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.477381    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.578954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.578901    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.579244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.579219    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg49m\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-kube-api-access-jg49m\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.579382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.579367    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.579539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.579238    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-credential-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.579630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.579533    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.579838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.579629    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-data\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.579915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.579819    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.579915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.579885    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.579915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.579822    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-certs\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.580104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.579996    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.580104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.580052    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a170-f2b9-4920-8939-5eb2e687e4df-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.580104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.580082    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.580104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.580091    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-socket\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.580610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.580586    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a170-f2b9-4920-8939-5eb2e687e4df-istiod-ca-cert\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.583509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.583481    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-podinfo\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.587872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.587852    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-envoy\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.592476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.592421    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg49m\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-kube-api-access-jg49m\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.595704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.594247    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-token\") pod \"e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") " pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.787775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.783919    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:36:32.892296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.889018    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:32.954731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.944513    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:36:32.954731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.944592    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 19:36:32.977982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.977949    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:36:32.978252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:32.978215    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:36:33.519616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:33.519123    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:33.779129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:33.763314    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"]
Oct 30 19:36:33.782872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:33.781191    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:33.782872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:33.781279    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:33.782872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:33.781319    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:33.982834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:33.975893    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:36:33.982834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:33.975961    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 19:36:33.982834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:33.978753    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:36:33.982834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:33.978809    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:36:34.351744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:34.351709    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" event={"ID":"34b7a170-f2b9-4920-8939-5eb2e687e4df","Type":"ContainerStarted","Data":"47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb"}
Oct 30 19:36:34.352065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:34.352028    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" event={"ID":"34b7a170-f2b9-4920-8939-5eb2e687e4df","Type":"ContainerStarted","Data":"72d927553bb9f25fd5399cce8c50d70b68c9a84bef995c4edd41a996a9bfbbc7"}
Oct 30 19:36:34.874733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:34.874700    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:34.889646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:34.889603    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:34.964711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:34.961889    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:36:34.964711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:34.961949    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 19:36:34.980717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:34.976872    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:36:34.980717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:34.976930    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:36:35.134045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.133902    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:35.882712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.881967    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" podStartSLOduration=3.8819415470000003 podStartE2EDuration="3.881941547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:34.391928239 +0000 UTC m=+1829.294941276" watchObservedRunningTime="2026-10-30 19:36:35.881941547 +0000 UTC m=+1830.784954589"
Oct 30 19:36:35.885713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.883519    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"]
Oct 30 19:36:35.885713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.884571    4925 cpu_manager.go:414] "containerMap: removing container" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy"
Oct 30 19:36:35.885713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.884588    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25918327-0d30-438b-979c-4531f0fd270f" containerName="istio-proxy"
Oct 30 19:36:35.904157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.900260    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"]
Oct 30 19:36:35.904157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.900376    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:35.972759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.971325    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 30 19:36:35.972759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.971391    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 30 19:36:35.972759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.971879    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:36:35.972759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.971920    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:36:36.008621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.985804    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:36:36.008621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:35.985853    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:36:36.019506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.019472    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f2dbd9-openshift-default-dockercfg-hl6bt\""
Oct 30 19:36:36.122025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.101805    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istiod-ca-cert\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.122025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.119560    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcr5q\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-kube-api-access-rcr5q\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.122025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.119711    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-credential-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.122025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.119751    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-podinfo\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.122025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.119798    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-token\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.122025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.119852    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-data\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.122025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.119912    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-envoy\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.122025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.120090    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-certs\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.122025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.120194    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.250100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.233137    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:36.294729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.292772    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-credential-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.295039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.295011    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-podinfo\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.298919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.298891    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-token\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.299132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.299111    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-data\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.299391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.299369    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-envoy\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.299662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.299640    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-certs\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.302887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.302861    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.312723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.303311    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istiod-ca-cert\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.356896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.356857    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcr5q\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-kube-api-access-rcr5q\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.391754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.313986    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-credential-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.392013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.315130    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-certs\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.430336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.313520    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-data\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.507773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.449338    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-socket\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.828821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.473987    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istiod-ca-cert\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.828821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.562331    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:36.828821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.675398    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"]
Oct 30 19:36:36.861895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.792320    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-token\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.880323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.880288    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"]
Oct 30 19:36:36.880753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.880731    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:36.971967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.971937    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0a6e66\"/\"default-dockercfg-xfs4c\""
Oct 30 19:36:36.983453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.983421    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0a6e66\"/\"auth-test-model-0a6e66-kserve-self-signed-certs\""
Oct 30 19:36:36.984235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.984211    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0a6e66\"/\"kube-root-ca.crt\""
Oct 30 19:36:36.985204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.985181    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0a6e66\"/\"openshift-service-ca.crt\""
Oct 30 19:36:36.994257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.994217    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-podinfo\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.996717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.994722    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcr5q\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-kube-api-access-rcr5q\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:36.997592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:36.997274    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-envoy\") pod \"e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") " pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:37.033978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.032036    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:36:37.033978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.032110    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:36:37.067746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.052988    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:37.075972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.075942    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff9d2a-d624-4ab3-a316-13d796122ee2-tls-certs\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.076218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.076199    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhxrx\" (UniqueName: \"kubernetes.io/projected/aeff9d2a-d624-4ab3-a316-13d796122ee2-kube-api-access-zhxrx\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.076523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.076507    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-dshm\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.076818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.076801    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-model-cache\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.078623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.078596    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-home\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.078894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.078848    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-tmp-dir\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.080809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.080766    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:37.168277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.168057    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 30 19:36:37.169418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.169381    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:37.171476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.171359    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.181660    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff9d2a-d624-4ab3-a316-13d796122ee2-tls-certs\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.181727    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhxrx\" (UniqueName: \"kubernetes.io/projected/aeff9d2a-d624-4ab3-a316-13d796122ee2-kube-api-access-zhxrx\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.181860    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-dshm\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.182674    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-model-cache\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.182065    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-model-cache\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.182865    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-home\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.183906    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-tmp-dir\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.184237    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-home\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.184316    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-tmp-dir\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.187175    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"]
Oct 30 19:36:37.188850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.188020    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy" containerID="cri-o://e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0" gracePeriod=30
Oct 30 19:36:37.227801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.189608    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-dshm\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.354121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.354086    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff9d2a-d624-4ab3-a316-13d796122ee2-tls-certs\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.378759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.367557    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhxrx\" (UniqueName: \"kubernetes.io/projected/aeff9d2a-d624-4ab3-a316-13d796122ee2-kube-api-access-zhxrx\") pod \"auth-test-model-0a6e66-kserve-798896b49c-j2dqj\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.390788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.375472    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:37.391045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.375858    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:37.472713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.472195    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:37.532709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.529045    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:36:37.532709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.530671    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:36:37.576382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.574498    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:36:37.576382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.574784    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:36:37.609823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.609790    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:36:37.863892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.863206    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:37.893152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.893105    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:36:37.893411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.893368    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:36:37.971442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.971408    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"]
Oct 30 19:36:37.976075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.976029    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:37.976164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.976125    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:37.976199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:37.976169    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:38.076007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.075981    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"]
Oct 30 19:36:38.079664 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:36:38.079315    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeff9d2a_d624_4ab3_a316_13d796122ee2.slice/crio-5a27f7b7d7382331cf03872b5cf8d3f90ec926a8163ecea9eed27f726f3933a4 WatchSource:0}: Error finding container 5a27f7b7d7382331cf03872b5cf8d3f90ec926a8163ecea9eed27f726f3933a4: Status 404 returned error can't find the container with id 5a27f7b7d7382331cf03872b5cf8d3f90ec926a8163ecea9eed27f726f3933a4
Oct 30 19:36:38.844715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.838473    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" event={"ID":"aeff9d2a-d624-4ab3-a316-13d796122ee2","Type":"ContainerStarted","Data":"9aebc5e13a5de9e2cb3a1d49a39298366603354d5ea8b7f109d0197877543178"}
Oct 30 19:36:38.844715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.838527    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" event={"ID":"aeff9d2a-d624-4ab3-a316-13d796122ee2","Type":"ContainerStarted","Data":"5a27f7b7d7382331cf03872b5cf8d3f90ec926a8163ecea9eed27f726f3933a4"}
Oct 30 19:36:38.854727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.853964    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" event={"ID":"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9","Type":"ContainerStarted","Data":"bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99"}
Oct 30 19:36:38.854727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.854028    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" event={"ID":"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9","Type":"ContainerStarted","Data":"7ba49453f2d91494dd99c952e48f1819109e60ff71ac171ce8d2aaec44105a21"}
Oct 30 19:36:38.868647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.865045    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:38.882375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.882314    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" podStartSLOduration=2.882294936 podStartE2EDuration="2.882294936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:38.872978331 +0000 UTC m=+1833.775991369" watchObservedRunningTime="2026-10-30 19:36:38.882294936 +0000 UTC m=+1833.785307979"
Oct 30 19:36:38.889762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.889729    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:36:38.889991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.889786    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:36:38.950805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:38.944131    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" podStartSLOduration=3.944105157 podStartE2EDuration="3.944105157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:38.907720559 +0000 UTC m=+1833.810733600" watchObservedRunningTime="2026-10-30 19:36:38.944105157 +0000 UTC m=+1833.847118195"
Oct 30 19:36:39.075210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:39.074746    4925 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:36:39.075210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:39.074827    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 19:36:39.075210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:39.075124    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:39.472929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:39.472896    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:39.840779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:39.832303    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:39.892775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:39.892226    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:36:39.892775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:39.892289    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:36:40.067130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:40.059279    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:40.091812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:40.086897    4925 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:36:40.091812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:40.086953    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 19:36:40.787718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:40.777806    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:40.892712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:40.892431    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:36:40.892712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:40.892517    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:36:41.067600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:41.067467    4925 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:36:41.067600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:41.067544    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 19:36:41.558718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:41.557872    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:41.890202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:41.890169    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:36:41.890401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:41.890219    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:36:42.067997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.067955    4925 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:36:42.068450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.068009    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 19:36:42.150446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.150350    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:42.638065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.638040    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:36:42.737630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.737601    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"]
Oct 30 19:36:42.738480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.738457    4925 cpu_manager.go:414] "containerMap: removing container" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy"
Oct 30 19:36:42.738480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.738476    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy"
Oct 30 19:36:42.738643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.738634    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerName="istio-proxy"
Oct 30 19:36:42.741173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.741157    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.745383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.744333    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:42.745383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.744836    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:36:42.751795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.751774    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"]
Oct 30 19:36:42.752533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.752514    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-envoy\") pod \"66a3e104-73aa-4209-a725-c333f0b7ecd8\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") "
Oct 30 19:36:42.752643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.752552    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-data\") pod \"66a3e104-73aa-4209-a725-c333f0b7ecd8\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") "
Oct 30 19:36:42.752643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.752585    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-credential-socket\") pod \"66a3e104-73aa-4209-a725-c333f0b7ecd8\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") "
Oct 30 19:36:42.752643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.752622    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-kube-api-access-6flhl\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-kube-api-access-6flhl\") pod \"66a3e104-73aa-4209-a725-c333f0b7ecd8\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") "
Oct 30 19:36:42.752947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.752711    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a3e104-73aa-4209-a725-c333f0b7ecd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a3e104-73aa-4209-a725-c333f0b7ecd8-istiod-ca-cert\") pod \"66a3e104-73aa-4209-a725-c333f0b7ecd8\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") "
Oct 30 19:36:42.752947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.752865    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-socket\") pod \"66a3e104-73aa-4209-a725-c333f0b7ecd8\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") "
Oct 30 19:36:42.752947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.752902    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-token\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-token\") pod \"66a3e104-73aa-4209-a725-c333f0b7ecd8\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") "
Oct 30 19:36:42.753145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.752955    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-podinfo\") pod \"66a3e104-73aa-4209-a725-c333f0b7ecd8\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") "
Oct 30 19:36:42.753145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.753027    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-certs\") pod \"66a3e104-73aa-4209-a725-c333f0b7ecd8\" (UID: \"66a3e104-73aa-4209-a725-c333f0b7ecd8\") "
Oct 30 19:36:42.754483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.754458    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-certs" pod "66a3e104-73aa-4209-a725-c333f0b7ecd8" (UID: "66a3e104-73aa-4209-a725-c333f0b7ecd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:42.755208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.755187    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a3e104-73aa-4209-a725-c333f0b7ecd8-istiod-ca-cert" pod "66a3e104-73aa-4209-a725-c333f0b7ecd8" (UID: "66a3e104-73aa-4209-a725-c333f0b7ecd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:42.755655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.755597    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-data" pod "66a3e104-73aa-4209-a725-c333f0b7ecd8" (UID: "66a3e104-73aa-4209-a725-c333f0b7ecd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:42.756211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.756192    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-socket" pod "66a3e104-73aa-4209-a725-c333f0b7ecd8" (UID: "66a3e104-73aa-4209-a725-c333f0b7ecd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:42.756830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.756702    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-credential-socket" pod "66a3e104-73aa-4209-a725-c333f0b7ecd8" (UID: "66a3e104-73aa-4209-a725-c333f0b7ecd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:42.758262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.758238    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-podinfo" pod "66a3e104-73aa-4209-a725-c333f0b7ecd8" (UID: "66a3e104-73aa-4209-a725-c333f0b7ecd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:42.758483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.758457    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-kube-api-access-6flhl" pod "66a3e104-73aa-4209-a725-c333f0b7ecd8" (UID: "66a3e104-73aa-4209-a725-c333f0b7ecd8"). InnerVolumeSpecName "kube-api-access-6flhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:42.760965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.760945    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-envoy" pod "66a3e104-73aa-4209-a725-c333f0b7ecd8" (UID: "66a3e104-73aa-4209-a725-c333f0b7ecd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:42.761052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.761000    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-token" pod "66a3e104-73aa-4209-a725-c333f0b7ecd8" (UID: "66a3e104-73aa-4209-a725-c333f0b7ecd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:42.855287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855200    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0109f6-4qk9n\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.855287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855234    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzm98\" (UniqueName: \"kubernetes.io/projected/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-kube-api-access-dzm98\") pod \"maas-api-revoke-sub-8e0109f6-4qk9n\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.855287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855277    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0109f6-4qk9n\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.855644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855625    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.855704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855649    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.855704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855658    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.855704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855667    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.855704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855676    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.855832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855726    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.855832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855737    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a3e104-73aa-4209-a725-c333f0b7ecd8-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.855832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855746    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6flhl\" (UniqueName: \"kubernetes.io/projected/66a3e104-73aa-4209-a725-c333f0b7ecd8-kube-api-access-6flhl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.855832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.855755    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a3e104-73aa-4209-a725-c333f0b7ecd8-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.919716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.917989    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:42.920038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.920000    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:42.921042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.921023    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:36:42.968875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.968839    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0109f6-4qk9n\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.969552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.969506    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0109f6-4qk9n\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.969777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.969557    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzm98\" (UniqueName: \"kubernetes.io/projected/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-kube-api-access-dzm98\") pod \"maas-api-revoke-sub-8e0109f6-4qk9n\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.971082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.971054    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0109f6-4qk9n\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.974959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.974932    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:42.976383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.976343    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0109f6-4qk9n\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.981290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.981268    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzm98\" (UniqueName: \"kubernetes.io/projected/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-kube-api-access-dzm98\") pod \"maas-api-revoke-sub-8e0109f6-4qk9n\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:42.985166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.985140    4925 generic.go:356] "Generic (PLEG): container finished" podID="66a3e104-73aa-4209-a725-c333f0b7ecd8" containerID="e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0" exitCode=0
Oct 30 19:36:42.985269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.985199    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"
Oct 30 19:36:42.985269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.985221    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" event={"ID":"66a3e104-73aa-4209-a725-c333f0b7ecd8","Type":"ContainerDied","Data":"e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0"}
Oct 30 19:36:42.985400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.985270    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z" event={"ID":"66a3e104-73aa-4209-a725-c333f0b7ecd8","Type":"ContainerDied","Data":"fdb9aaaeb80f30106ec9b2de01909e1922e36d21385169f05595cdb065a6fb6a"}
Oct 30 19:36:42.985400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:42.985291    4925 scope.go:122] "RemoveContainer" containerID="e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0"
Oct 30 19:36:43.060511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.058808    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"]
Oct 30 19:36:43.068342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.068263    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17e904-openshift-default-7cfdf6f655-sjn2z"]
Oct 30 19:36:43.068956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.068413    4925 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:36:43.068956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.068455    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 19:36:43.080647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.080302    4925 scope.go:122] "RemoveContainer" containerID="e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0"
Oct 30 19:36:43.080789 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:36:43.080723    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0\": container with ID starting with e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0 not found: ID does not exist" containerID="e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0"
Oct 30 19:36:43.080789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.080758    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0"} err="failed to get container status \"e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0\": rpc error: code = NotFound desc = could not find container \"e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0\": container with ID starting with e2ce48e3b258469a0d23ccbc6811d6611aea8933e4bd85450cd4ef42746f61f0 not found: ID does not exist"
Oct 30 19:36:43.084952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.084710    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:43.267018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.266985    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a3e104-73aa-4209-a725-c333f0b7ecd8" path="/var/lib/kubelet/pods/66a3e104-73aa-4209-a725-c333f0b7ecd8/volumes"
Oct 30 19:36:43.428564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.428532    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"]
Oct 30 19:36:43.431647 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:36:43.431620    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a5e0b3_69d1_4c92_981d_c6e8526a98b4.slice/crio-18f58f95be744a9584b00ac50d717cc9c4aab665d8ce9f9a547240a8ff7bff8d WatchSource:0}: Error finding container 18f58f95be744a9584b00ac50d717cc9c4aab665d8ce9f9a547240a8ff7bff8d: Status 404 returned error can't find the container with id 18f58f95be744a9584b00ac50d717cc9c4aab665d8ce9f9a547240a8ff7bff8d
Oct 30 19:36:43.751769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:43.750520    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:44.025245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:44.025128    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n" event={"ID":"30a5e0b3-69d1-4c92-981d-c6e8526a98b4","Type":"ContainerStarted","Data":"4f06786c16927ed2cc66a587f33012b190499afe4fcc41ba4ef906e0bfec4c2b"}
Oct 30 19:36:44.025245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:44.025189    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n" event={"ID":"30a5e0b3-69d1-4c92-981d-c6e8526a98b4","Type":"ContainerStarted","Data":"18f58f95be744a9584b00ac50d717cc9c4aab665d8ce9f9a547240a8ff7bff8d"}
Oct 30 19:36:44.045602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:44.045556    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n" podStartSLOduration=2.04554293 podStartE2EDuration="2.04554293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:44.040821066 +0000 UTC m=+1838.943834103" watchObservedRunningTime="2026-10-30 19:36:44.04554293 +0000 UTC m=+1838.948555967"
Oct 30 19:36:44.067843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:44.067811    4925 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:36:44.067989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:44.067871    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 30 19:36:44.550830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:44.550748    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:45.038744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:45.038708    4925 generic.go:356] "Generic (PLEG): container finished" podID="30a5e0b3-69d1-4c92-981d-c6e8526a98b4" containerID="4f06786c16927ed2cc66a587f33012b190499afe4fcc41ba4ef906e0bfec4c2b" exitCode=0
Oct 30 19:36:45.038941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:45.038783    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n" event={"ID":"30a5e0b3-69d1-4c92-981d-c6e8526a98b4","Type":"ContainerDied","Data":"4f06786c16927ed2cc66a587f33012b190499afe4fcc41ba4ef906e0bfec4c2b"}
Oct 30 19:36:45.072044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:45.072017    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:45.072392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:45.072366    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:45.073006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:45.072985    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:36:45.321114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:45.321017    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:45.966431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:45.962473    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:46.221099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.221032    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:46.232436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.232412    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-xks-ca\") pod \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") "
Oct 30 19:36:46.232568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.232515    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-openshift-ca\") pod \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") "
Oct 30 19:36:46.232709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.232672    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-kube-api-access-dzm98\" (UniqueName: \"kubernetes.io/projected/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-kube-api-access-dzm98\") pod \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\" (UID: \"30a5e0b3-69d1-4c92-981d-c6e8526a98b4\") "
Oct 30 19:36:46.232948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.232920    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-openshift-ca" pod "30a5e0b3-69d1-4c92-981d-c6e8526a98b4" (UID: "30a5e0b3-69d1-4c92-981d-c6e8526a98b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:46.233460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.233436    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:46.234820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.234778    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-xks-ca" pod "30a5e0b3-69d1-4c92-981d-c6e8526a98b4" (UID: "30a5e0b3-69d1-4c92-981d-c6e8526a98b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:46.235041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.235019    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-kube-api-access-dzm98" pod "30a5e0b3-69d1-4c92-981d-c6e8526a98b4" (UID: "30a5e0b3-69d1-4c92-981d-c6e8526a98b4"). InnerVolumeSpecName "kube-api-access-dzm98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:46.335140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.335107    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzm98\" (UniqueName: \"kubernetes.io/projected/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-kube-api-access-dzm98\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:46.335140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.335136    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a5e0b3-69d1-4c92-981d-c6e8526a98b4-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:46.831773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:46.826472    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:47.056861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:47.056832    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"
Oct 30 19:36:47.057311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:47.056871    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n" event={"ID":"30a5e0b3-69d1-4c92-981d-c6e8526a98b4","Type":"ContainerDied","Data":"18f58f95be744a9584b00ac50d717cc9c4aab665d8ce9f9a547240a8ff7bff8d"}
Oct 30 19:36:47.057311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:47.056927    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f58f95be744a9584b00ac50d717cc9c4aab665d8ce9f9a547240a8ff7bff8d"
Oct 30 19:36:47.473637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:47.473607    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:47.486909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:47.486886    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:47.548877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:47.548639    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:48.076392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.076361    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:36:48.144989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.144942    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:48.552878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.552844    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"]
Oct 30 19:36:48.553552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.553538    4925 cpu_manager.go:414] "containerMap: removing container" podUID="30a5e0b3-69d1-4c92-981d-c6e8526a98b4" containerName="revoke-keys"
Oct 30 19:36:48.553599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.553553    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a5e0b3-69d1-4c92-981d-c6e8526a98b4" containerName="revoke-keys"
Oct 30 19:36:48.553721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.553680    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a5e0b3-69d1-4c92-981d-c6e8526a98b4" containerName="revoke-keys"
Oct 30 19:36:48.556238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.556223    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.559322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.559294    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1fa500-openshift-default-dockercfg-vvhvs\""
Oct 30 19:36:48.568734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.568676    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"]
Oct 30 19:36:48.677077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.677044    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.677257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.677204    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-credential-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.677298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.677255    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82pcl\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-kube-api-access-82pcl\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.677335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.677301    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-envoy\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.677407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.677386    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-data\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.677563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.677549    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942788f9-4ccc-4fec-917a-3ff94573071c-istio-podinfo\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.677674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.677663    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-istio-token\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.677755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.677743    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942788f9-4ccc-4fec-917a-3ff94573071c-istiod-ca-cert\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.677828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.677811    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-certs\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.779845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.779812    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942788f9-4ccc-4fec-917a-3ff94573071c-istiod-ca-cert\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.779853    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-certs\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.779893    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.779949    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-credential-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.779975    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82pcl\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-kube-api-access-82pcl\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.780005    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-envoy\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.780026    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-data\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.780193    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942788f9-4ccc-4fec-917a-3ff94573071c-istio-podinfo\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.780331    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-istio-token\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.780376    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.780416    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-credential-socket\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.780515    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-certs\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.780598    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-data\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.780654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.780600    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942788f9-4ccc-4fec-917a-3ff94573071c-istiod-ca-cert\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.782581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.782556    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-envoy\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.783074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.783053    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942788f9-4ccc-4fec-917a-3ff94573071c-istio-podinfo\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.791812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.791789    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-istio-token\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.791955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.791933    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82pcl\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-kube-api-access-82pcl\") pod \"e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") " pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.870654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.870574    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:48.979452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:48.979276    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:49.205813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:49.205783    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"]
Oct 30 19:36:49.206981 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:36:49.206954    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942788f9_4ccc_4fec_917a_3ff94573071c.slice/crio-dabe85d8bdfe5c8f8a5c5f23cac4efedd3cdcb518141d026085667da498d93c6 WatchSource:0}: Error finding container dabe85d8bdfe5c8f8a5c5f23cac4efedd3cdcb518141d026085667da498d93c6: Status 404 returned error can't find the container with id dabe85d8bdfe5c8f8a5c5f23cac4efedd3cdcb518141d026085667da498d93c6
Oct 30 19:36:49.210372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:49.210334    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:49.210504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:49.210419    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:49.210504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:49.210472    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:36:49.570424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:49.570288    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:50.085555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:50.085515    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" event={"ID":"942788f9-4ccc-4fec-917a-3ff94573071c","Type":"ContainerStarted","Data":"7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40"}
Oct 30 19:36:50.086459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:50.085587    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" event={"ID":"942788f9-4ccc-4fec-917a-3ff94573071c","Type":"ContainerStarted","Data":"dabe85d8bdfe5c8f8a5c5f23cac4efedd3cdcb518141d026085667da498d93c6"}
Oct 30 19:36:50.111028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:50.109519    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" podStartSLOduration=2.109505093 podStartE2EDuration="2.109505093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:50.108070738 +0000 UTC m=+1845.011083778" watchObservedRunningTime="2026-10-30 19:36:50.109505093 +0000 UTC m=+1845.012518127"
Oct 30 19:36:50.541790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:50.534392    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:50.870861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:50.870781    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:50.872409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:50.872380    4925 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:36:50.872573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:50.872441    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:36:51.374825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:51.374791    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:51.743917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:51.743869    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:51.873905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:51.872131    4925 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:36:51.873905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:51.872201    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:36:52.182756 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:36:52.182643    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:36:52.276002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.275963    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:52.871175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.871132    4925 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:36:52.872305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.872263    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:36:52.956643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.956574    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"]
Oct 30 19:36:52.965925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.965903    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:52.967350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.967326    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"]
Oct 30 19:36:52.975428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.974958    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ddb6ea\"/\"kube-root-ca.crt\""
Oct 30 19:36:52.975428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.975033    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ddb6ea\"/\"openshift-service-ca.crt\""
Oct 30 19:36:52.975428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.975169    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ddb6ea\"/\"auth-test-model-ddb6ea-kserve-self-signed-certs\""
Oct 30 19:36:52.975428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:52.975262    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ddb6ea\"/\"default-dockercfg-qlgfh\""
Oct 30 19:36:53.030759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.014894    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-dshm\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.031103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.031068    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-model-cache\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.035905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.035875    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-home\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.036099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.036079    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152df259-957c-42b1-915c-cc6b35b37e7b-tls-certs\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.048707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.042953    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-tmp-dir\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.048707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.043138    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7w9z\" (UniqueName: \"kubernetes.io/projected/152df259-957c-42b1-915c-cc6b35b37e7b-kube-api-access-m7w9z\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.051972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.051836    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:53.171711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.148411    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-tmp-dir\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.172200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.172167    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7w9z\" (UniqueName: \"kubernetes.io/projected/152df259-957c-42b1-915c-cc6b35b37e7b-kube-api-access-m7w9z\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.172570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.172549    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-dshm\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.172734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.172716    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-model-cache\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.179521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.172968    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-home\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.179753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.179728    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152df259-957c-42b1-915c-cc6b35b37e7b-tls-certs\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.180081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.176648    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-model-cache\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.180299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.176906    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-home\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.186755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.163535    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-tmp-dir\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.244714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.217076    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-dshm\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.269049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.244889    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7w9z\" (UniqueName: \"kubernetes.io/projected/152df259-957c-42b1-915c-cc6b35b37e7b-kube-api-access-m7w9z\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.269049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.227579    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:53.273712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.273305    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152df259-957c-42b1-915c-cc6b35b37e7b-tls-certs\") pod \"auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") " pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.320714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.304728    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:53.831593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.830071    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"]
Oct 30 19:36:53.870957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.870928    4925 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:36:53.871076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:53.870999    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:36:54.028415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:54.028379    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:54.177797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:54.177755    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" event={"ID":"152df259-957c-42b1-915c-cc6b35b37e7b","Type":"ContainerStarted","Data":"fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674"}
Oct 30 19:36:54.178004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:54.177819    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" event={"ID":"152df259-957c-42b1-915c-cc6b35b37e7b","Type":"ContainerStarted","Data":"30d7a0e6ca562b8ded1d3e132faacf4eee3515ba5900c8abccfceff2dffdb442"}
Oct 30 19:36:54.230221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:54.230165    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podStartSLOduration=2.230149736 podStartE2EDuration="2.230149736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:54.226782829 +0000 UTC m=+1849.129795866" watchObservedRunningTime="2026-10-30 19:36:54.230149736 +0000 UTC m=+1849.133162766"
Oct 30 19:36:54.260938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:54.260901    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:54.350746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:54.344444    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:36:54.871018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:54.870977    4925 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:36:54.871216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:54.871041    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:36:54.871394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:54.871367    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:55.043715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:55.043372    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:55.861396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:55.858268    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:55.876708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:55.871543    4925 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:36:55.876708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:55.871647    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:36:56.569298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:56.569253    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:56.876316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:56.876236    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:56.876744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:56.876718    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:56.877444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:56.877425    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:36:57.383730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:57.383091    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:57.691870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:57.691827    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:58.266714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:58.261924    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:58.852525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:58.852479    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:59.588634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:36:59.588578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:00.111038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:00.110997    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"]
Oct 30 19:37:00.205099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:00.205048    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:00.970780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:00.970706    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:02.464197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:02.464164    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:02.654710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:02.650001    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:02.811179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:02.811073    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:03.262640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:03.262579    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:03.342864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:03.342833    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:37:03.356540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:03.356515    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:37:03.373581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:03.373545    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:37:04.160219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:04.160184    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:05.066446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:05.066414    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:05.769980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:05.769942    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:06.880594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:06.880553    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:07.653454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:07.653415    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:08.465595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:08.465466    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:09.372650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:09.372614    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:09.814315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:09.814288    4925 scope.go:122] "RemoveContainer" containerID="df4dafa7b8310dfd9603114343bc15b64b22d2d08d4ea67e86a2140ac3c2fac2"
Oct 30 19:37:10.081717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:10.081498    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:10.696714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:10.696092    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:11.480018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:11.479969    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:12.158115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:12.154088    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:12.919308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:12.919270    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:13.088283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:13.088249    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:13.664186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:13.660445    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:14.281361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:14.281325    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:15.144594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.144555    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f" podUID="ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" containerName="main" containerID="cri-o://c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd" gracePeriod=45
Oct 30 19:37:15.268511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.268481    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:15.465161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.465137    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:37:15.512562 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.512524    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tmp-dir\") pod \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") "
Oct 30 19:37:15.512760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.512646    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tls-certs\") pod \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") "
Oct 30 19:37:15.512760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.512719    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-model-cache\") pod \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") "
Oct 30 19:37:15.512907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.512782    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tmp-dir" pod "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" (UID: "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:15.512907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.512846    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-dshm\") pod \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") "
Oct 30 19:37:15.512907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.512901    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-home\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-home\") pod \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") "
Oct 30 19:37:15.513435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.513008    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-kube-api-access-gthqn\" (UniqueName: \"kubernetes.io/projected/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-kube-api-access-gthqn\") pod \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\" (UID: \"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1\") "
Oct 30 19:37:15.513593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.513029    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-model-cache" pod "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" (UID: "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:15.513593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.513236    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-home" pod "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" (UID: "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:15.514399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.514377    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:15.514399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.514399    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:15.514573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.514409    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:15.516187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.516164    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-dshm" pod "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" (UID: "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:15.516303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.516258    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tls-certs" pod "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" (UID: "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:15.516636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.516609    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-kube-api-access-gthqn" pod "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" (UID: "ed2fd33e-eed8-40e5-8a3e-7a62db8412c1"). InnerVolumeSpecName "kube-api-access-gthqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:15.531973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.531943    4925 generic.go:356] "Generic (PLEG): container finished" podID="ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" containerID="c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd" exitCode=0
Oct 30 19:37:15.532136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.532023    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"
Oct 30 19:37:15.532136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.532038    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f" event={"ID":"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1","Type":"ContainerDied","Data":"c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd"}
Oct 30 19:37:15.532136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.532087    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f" event={"ID":"ed2fd33e-eed8-40e5-8a3e-7a62db8412c1","Type":"ContainerDied","Data":"7f28bfaf6fb3d3aabf75a712ee339a7e65cd5bd3e359aa8c48808c9701e28b11"}
Oct 30 19:37:15.532136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.532110    4925 scope.go:122] "RemoveContainer" containerID="c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd"
Oct 30 19:37:15.556464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.556443    4925 scope.go:122] "RemoveContainer" containerID="c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd"
Oct 30 19:37:15.556825 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:37:15.556804    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd\": container with ID starting with c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd not found: ID does not exist" containerID="c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd"
Oct 30 19:37:15.556903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.556834    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd"} err="failed to get container status \"c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd\": rpc error: code = NotFound desc = could not find container \"c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd\": container with ID starting with c28af309870560d1de0ac7fcc2ce2d376c53fe6a0b9ab82b9b0cd63b5ec04acd not found: ID does not exist"
Oct 30 19:37:15.567357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.567335    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"]
Oct 30 19:37:15.576099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.575491    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c24c01/test-model-c24c01-kserve-f5554cd8f-d8b8f"]
Oct 30 19:37:15.616305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.616271    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:15.616305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.616302    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:15.616471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:15.616312    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gthqn\" (UniqueName: \"kubernetes.io/projected/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1-kube-api-access-gthqn\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:16.005277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:16.005230    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:16.639714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:16.637783    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:17.264968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:17.264925    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" path="/var/lib/kubelet/pods/ed2fd33e-eed8-40e5-8a3e-7a62db8412c1/volumes"
Oct 30 19:37:17.574784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:17.574669    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:18.366153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:18.366113    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:18.600457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:18.600419    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:19.283988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:19.283958    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:19.879283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:19.879233    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:20.193464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.193419    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"]
Oct 30 19:37:20.195071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.195044    4925 cpu_manager.go:414] "containerMap: removing container" podUID="ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" containerName="main"
Oct 30 19:37:20.195071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.195072    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" containerName="main"
Oct 30 19:37:20.195557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.195536    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2fd33e-eed8-40e5-8a3e-7a62db8412c1" containerName="main"
Oct 30 19:37:20.200881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.200857    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.208825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.208803    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-fd7c77-openshift-default-dockercfg-9t69p\""
Oct 30 19:37:20.210612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.210590    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"]
Oct 30 19:37:20.343561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.343526    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.343765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.343597    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.343835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.343810    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.343895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.343839    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3301076-3aef-468a-a79a-04c0ca49231c-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.343996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.343973    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rb4d\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-kube-api-access-8rb4d\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.344055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.344043    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3301076-3aef-468a-a79a-04c0ca49231c-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.344166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.344155    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.344212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.344197    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.344263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.344221    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446120    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rb4d\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-kube-api-access-8rb4d\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446202    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3301076-3aef-468a-a79a-04c0ca49231c-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446267    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446314    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446338    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446399    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446463    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446547    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446576    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3301076-3aef-468a-a79a-04c0ca49231c-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446780    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.446827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.446816    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.447357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.447052    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.447357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.447084    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.447357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.447085    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3301076-3aef-468a-a79a-04c0ca49231c-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.448976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.448953    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.449246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.449227    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3301076-3aef-468a-a79a-04c0ca49231c-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.457905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.457877    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rb4d\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-kube-api-access-8rb4d\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.457905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.457898    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.547764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.547729    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:20.680314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.680280    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:20.850259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.850217    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"]
Oct 30 19:37:20.851542 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:37:20.851493    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3301076_3aef_468a_a79a_04c0ca49231c.slice/crio-5c5e6451ccb877ca60150cb53b74a58f83deeb53b649bd2d87c4326fa93324ff WatchSource:0}: Error finding container 5c5e6451ccb877ca60150cb53b74a58f83deeb53b649bd2d87c4326fa93324ff: Status 404 returned error can't find the container with id 5c5e6451ccb877ca60150cb53b74a58f83deeb53b649bd2d87c4326fa93324ff
Oct 30 19:37:20.854053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.853949    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:20.855320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.855287    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:20.855444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.855372    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:20.855444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:20.855417    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:21.583842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:21.583811    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:21.611279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:21.611242    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" event={"ID":"c3301076-3aef-468a-a79a-04c0ca49231c","Type":"ContainerStarted","Data":"daf390c4986197c06ea34f1cb050a64218a7e1892d5e0cef107e658e275c51bc"}
Oct 30 19:37:21.611448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:21.611303    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" event={"ID":"c3301076-3aef-468a-a79a-04c0ca49231c","Type":"ContainerStarted","Data":"5c5e6451ccb877ca60150cb53b74a58f83deeb53b649bd2d87c4326fa93324ff"}
Oct 30 19:37:21.639529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:21.639470    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" podStartSLOduration=1.639449288 podStartE2EDuration="1.639449288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:21.633063871 +0000 UTC m=+1876.536076909" watchObservedRunningTime="2026-10-30 19:37:21.639449288 +0000 UTC m=+1876.542462326"
Oct 30 19:37:22.140316 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:37:22.140283    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:37:22.478634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:22.474053    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:22.548495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:22.548465    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:22.550369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:22.550343    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:37:22.550505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:22.550390    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:37:23.286753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:23.286674    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:23.548932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:23.548857    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:37:23.548932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:23.548904    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:37:24.188076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:24.185831    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:24.549128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:24.549044    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:37:24.549519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:24.549115    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:37:24.865013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:24.864618    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:25.548090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:25.548054    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:37:25.548276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:25.548121    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:37:25.782712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:25.780062    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:26.467946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:26.466222    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:26.548857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:26.548814    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:37:26.549031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:26.548865    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:37:26.991567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:26.991534    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:27.157493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:27.157439    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:27.327769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:27.321213    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:27.582791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:27.562808    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:37:27.582791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:27.563025    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:37:28.071213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.069983    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:28.272033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.272000    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"]
Oct 30 19:37:28.276626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.276604    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.284509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.283881    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-2405fb-openshift-default-dockercfg-mpx47\""
Oct 30 19:37:28.290777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.290755    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"]
Oct 30 19:37:28.377635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.377549    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvldq\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-kube-api-access-pvldq\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.377635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.377593    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.377860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.377769    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.377860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.377787    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.377860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.377841    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.377971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.377940    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.378023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.378008    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.378141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.378120    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.378217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.378174    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.479743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.479663    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.479925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.479760    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.479925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.479844    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.479925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.479905    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.479977    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.480024    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.480064    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.480066    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.480271    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.480304    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvldq\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-kube-api-access-pvldq\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.480336    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.480346    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.480392    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.480828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.480798    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.482677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.482649    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.482916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.482897    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.491610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.491590    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.492046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.492028    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvldq\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-kube-api-access-pvldq\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.553462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.553434    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:28.553922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.553904    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:28.554586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.554564    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:37:28.596580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.595537    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:28.798625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.797551    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:28.912791 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:37:28.912761    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f17c29_b6ba_44b4_beb6_ac220cb960f3.slice/crio-bae077e002c0a2c429b44779e3b5bf6cb7121922f61b325709d442ba1f947a2c WatchSource:0}: Error finding container bae077e002c0a2c429b44779e3b5bf6cb7121922f61b325709d442ba1f947a2c: Status 404 returned error can't find the container with id bae077e002c0a2c429b44779e3b5bf6cb7121922f61b325709d442ba1f947a2c
Oct 30 19:37:28.914218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.914192    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"]
Oct 30 19:37:28.916133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.916100    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:28.916245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.916167    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:28.916245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:28.916195    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:29.430618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.430586    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:29.661469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.661429    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"]
Oct 30 19:37:29.667452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.667428    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.681515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.681459    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8e6762-openshift-default-dockercfg-ns2mw\""
Oct 30 19:37:29.688945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.688918    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"]
Oct 30 19:37:29.689904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.689871    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" event={"ID":"99f17c29-b6ba-44b4-beb6-ac220cb960f3","Type":"ContainerStarted","Data":"e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f"}
Oct 30 19:37:29.690011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.689931    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" event={"ID":"99f17c29-b6ba-44b4-beb6-ac220cb960f3","Type":"ContainerStarted","Data":"bae077e002c0a2c429b44779e3b5bf6cb7121922f61b325709d442ba1f947a2c"}
Oct 30 19:37:29.769435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.769380    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podStartSLOduration=1.769364527 podStartE2EDuration="1.769364527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:29.765631111 +0000 UTC m=+1884.668644154" watchObservedRunningTime="2026-10-30 19:37:29.769364527 +0000 UTC m=+1884.672377564"
Oct 30 19:37:29.829826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.829782    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.830182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.830129    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.830677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.830653    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rj82\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-kube-api-access-2rj82\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.831854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.831796    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.832029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.832010    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27748f04-aa17-4148-bfca-197df6076bab-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.833128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.833109    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.833362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.833343    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.834831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.834811    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.835155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.835133    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27748f04-aa17-4148-bfca-197df6076bab-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.936946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.936859    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.937198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.937172    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.937373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.937346    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.937577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.937555    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.937773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.937754    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27748f04-aa17-4148-bfca-197df6076bab-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.937942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.937925    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.938086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.938067    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.938269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.938230    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rj82\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-kube-api-access-2rj82\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.938432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.938414    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.938507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.938454    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27748f04-aa17-4148-bfca-197df6076bab-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.938724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.938703    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.942132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.938977    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.942132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.939325    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27748f04-aa17-4148-bfca-197df6076bab-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.942132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.939496    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.942132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.940927    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.942132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.941945    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27748f04-aa17-4148-bfca-197df6076bab-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.950428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.950401    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.950917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.950897    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rj82\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-kube-api-access-2rj82\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:29.985347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:29.985318    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:30.330794 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:37:30.330759    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27748f04_aa17_4148_bfca_197df6076bab.slice/crio-a1a6a6fea17eb5e0d1abdd7a6c4351d3fe6623c5d198bdc0b41bf496c23a5fa7 WatchSource:0}: Error finding container a1a6a6fea17eb5e0d1abdd7a6c4351d3fe6623c5d198bdc0b41bf496c23a5fa7: Status 404 returned error can't find the container with id a1a6a6fea17eb5e0d1abdd7a6c4351d3fe6623c5d198bdc0b41bf496c23a5fa7
Oct 30 19:37:30.330991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.330968    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"]
Oct 30 19:37:30.334770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.334737    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:30.334860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.334820    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:30.334899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.334859    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:30.511715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.508084    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:30.599757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.596751    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:30.609713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.607204    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:37:30.609713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.607256    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:37:30.723449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.723402    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" event={"ID":"27748f04-aa17-4148-bfca-197df6076bab","Type":"ContainerStarted","Data":"c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd"}
Oct 30 19:37:30.723739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.723714    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" event={"ID":"27748f04-aa17-4148-bfca-197df6076bab","Type":"ContainerStarted","Data":"a1a6a6fea17eb5e0d1abdd7a6c4351d3fe6623c5d198bdc0b41bf496c23a5fa7"}
Oct 30 19:37:30.770058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.764009    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podStartSLOduration=1.763989901 podStartE2EDuration="1.763989901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:30.757002031 +0000 UTC m=+1885.660015069" watchObservedRunningTime="2026-10-30 19:37:30.763989901 +0000 UTC m=+1885.667002940"
Oct 30 19:37:30.985743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:30.985708    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:31.158738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.157568    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:31.599714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.599060    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:37:31.599714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.599135    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:37:31.915266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.915234    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:31.956076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.956036    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"]
Oct 30 19:37:31.979975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.979944    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:31.987878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.987833    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"]
Oct 30 19:37:31.990520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.990470    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:37:31.990663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.990564    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 19:37:31.993919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.990990    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd7c77\"/\"default-dockercfg-d4txb\""
Oct 30 19:37:31.993919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.991078    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd7c77\"/\"auth-test-model-fd7c77-kserve-self-signed-certs\""
Oct 30 19:37:31.993919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.991143    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd7c77\"/\"kube-root-ca.crt\""
Oct 30 19:37:31.993919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:31.991195    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd7c77\"/\"openshift-service-ca.crt\""
Oct 30 19:37:32.090282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.089742    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:32.122797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.121383    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-home\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.122797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.121516    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e106ffdc-325c-419f-ae77-5d51a742c185-tls-certs\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.122797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.121573    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-tmp-dir\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.122797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.121646    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-dshm\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.122797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.121828    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-model-cache\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.122797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.121872    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptt88\" (UniqueName: \"kubernetes.io/projected/e106ffdc-325c-419f-ae77-5d51a742c185-kube-api-access-ptt88\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.248768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.223727    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-model-cache\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.249120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.245431    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-model-cache\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.266714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.249295    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptt88\" (UniqueName: \"kubernetes.io/projected/e106ffdc-325c-419f-ae77-5d51a742c185-kube-api-access-ptt88\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.344079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.344024    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-home\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.372712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.349944    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e106ffdc-325c-419f-ae77-5d51a742c185-tls-certs\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.373139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.373107    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-tmp-dir\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.373371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.373355    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-dshm\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.406862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.354124    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-home\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.467231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.375116    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-tmp-dir\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.467523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.415616    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e106ffdc-325c-419f-ae77-5d51a742c185-tls-certs\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.473767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.471833    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptt88\" (UniqueName: \"kubernetes.io/projected/e106ffdc-325c-419f-ae77-5d51a742c185-kube-api-access-ptt88\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.492709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.485545    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-dshm\") pod \"auth-test-model-fd7c77-kserve-8696ff8869-zltp5\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") " pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:32.653045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.634213    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:37:32.672741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.668110    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:37:32.681570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:32.679075    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:33.004296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:33.004256    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:37:33.004586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:33.004556    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 19:37:33.010141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:33.008969    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:33.356631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:33.356601    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"]
Oct 30 19:37:33.357188 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:37:33.357164    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode106ffdc_325c_419f_ae77_5d51a742c185.slice/crio-4c608e047e59e16ed05bf72374789eefbe1dd5a2d5e4b064fdd5b22646fdbe66 WatchSource:0}: Error finding container 4c608e047e59e16ed05bf72374789eefbe1dd5a2d5e4b064fdd5b22646fdbe66: Status 404 returned error can't find the container with id 4c608e047e59e16ed05bf72374789eefbe1dd5a2d5e4b064fdd5b22646fdbe66
Oct 30 19:37:33.597574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:33.597541    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:37:33.597884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:33.597853    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:37:33.835569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:33.835533    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" event={"ID":"e106ffdc-325c-419f-ae77-5d51a742c185","Type":"ContainerStarted","Data":"05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c"}
Oct 30 19:37:33.836122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:33.836099    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" event={"ID":"e106ffdc-325c-419f-ae77-5d51a742c185","Type":"ContainerStarted","Data":"4c608e047e59e16ed05bf72374789eefbe1dd5a2d5e4b064fdd5b22646fdbe66"}
Oct 30 19:37:33.883762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:33.882417    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podStartSLOduration=2.882396455 podStartE2EDuration="2.882396455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:33.877138848 +0000 UTC m=+1888.780151910" watchObservedRunningTime="2026-10-30 19:37:33.882396455 +0000 UTC m=+1888.785409499"
Oct 30 19:37:34.037761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.011270    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:37:34.038147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.038108    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 19:37:34.192350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.192080    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:34.458372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.458285    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"]
Oct 30 19:37:34.464225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.464201    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.465167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.465142    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"]
Oct 30 19:37:34.471933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.471908    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"]
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.480546    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.480663    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxsj\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-kube-api-access-fdxsj\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.480748    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56680a43-73b9-4d8f-865d-e112f76bfb19-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.480817    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.480861    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.480906    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.480995    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.481042    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.481072    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.484719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.480994    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-69c573-openshift-default-dockercfg-5g2t8\""
Oct 30 19:37:34.485370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.485287    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"]
Oct 30 19:37:34.485439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.485376    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.582155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582074    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.582155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582121    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.582376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582198    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.582376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582273    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.582376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582307    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.582477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582429    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxsj\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-kube-api-access-fdxsj\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.582514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582472    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6ec1b5-8f21-429e-a7fc-be52be068305-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.582548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582534    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56680a43-73b9-4d8f-865d-e112f76bfb19-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582594    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582651    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582707    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4vr5\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-kube-api-access-f4vr5\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582758    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582806    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582833    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582858    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.582864    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.583030    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.583588    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.583732    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56680a43-73b9-4d8f-865d-e112f76bfb19-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.583858    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.583902    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.584059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.583943    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.584582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.584255    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.586114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.586061    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.586329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.586313    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.599725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.595157    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.599725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.595754    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxsj\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-kube-api-access-fdxsj\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.654711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.653793    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:37:34.654711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.653859    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:37:34.681030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.680971    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.686014    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.686100    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.686232    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6ec1b5-8f21-429e-a7fc-be52be068305-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.686660    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.686735    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4vr5\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-kube-api-access-f4vr5\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.686894    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.686915    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.687057    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.687083    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.687237    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.687745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.687619    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6ec1b5-8f21-429e-a7fc-be52be068305-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.688427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.687797    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.688427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.687874    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.688427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.687969    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.690539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.690516    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.691135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.691094    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.710761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.708217    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.714709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.712288    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4vr5\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-kube-api-access-f4vr5\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:34.791117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.791089    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:34.867126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:34.867094    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:35.024262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.024187    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:37:35.024530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.024460    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 19:37:35.071349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.071234    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:35.459886 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.456268    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"]
Oct 30 19:37:35.695714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.695243    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.695714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.695270    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:37:35.695714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.695327    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:37:35.716982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.716939    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:35.717212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.717191    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:35.717346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.717331    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:35.797711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.784353    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.797711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.784407    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.797711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.784456    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.797711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.784713    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.797711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.784760    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e4d4e-8cea-4f60-8061-2c5276c6e800-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.797711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.784846    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvl8\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-kube-api-access-dcvl8\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.797711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.784885    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.797711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.784932    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.797711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.784964    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.855594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.852409    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"]
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.892970    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.893039    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e4d4e-8cea-4f60-8061-2c5276c6e800-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.893129    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvl8\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-kube-api-access-dcvl8\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.893171    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.893219    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.893262    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.893372    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.893398    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.893445    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.899948    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-data\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.900603    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e4d4e-8cea-4f60-8061-2c5276c6e800-istiod-ca-cert\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.901048    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-credential-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.909740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.901277    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-certs\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:35.933746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:35.930682    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-socket\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:36.026780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.016808    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:37:36.026780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.016872    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 19:37:36.126335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.103739    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"]
Oct 30 19:37:36.126335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.118070    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-token\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:36.178096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.132408    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvl8\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-kube-api-access-dcvl8\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:36.178096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.140264    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:36.178096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.175774    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Oct 30 19:37:36.178096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.176499    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-envoy\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:36.211740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.194966    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:37:36.211740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.197303    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-podinfo\") pod \"e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") " pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:36.220714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.216199    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:36.331776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.245457    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"]
Oct 30 19:37:36.331776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.246079    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" event={"ID":"56680a43-73b9-4d8f-865d-e112f76bfb19","Type":"ContainerStarted","Data":"39ea6216b4ffca2f16dcbc761933db3145e4898b4fbc6f66401a6d30ae806424"}
Oct 30 19:37:36.331776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.249788    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:36.331776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.249876    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:36.331776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.249922    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:36.347136 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:37:36.343108    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6ec1b5_8f21_429e_a7fc_be52be068305.slice/crio-210463e3823a423c3f9d7804e66d15e8e45727239cb08c9eeac26aa5befc4d2c WatchSource:0}: Error finding container 210463e3823a423c3f9d7804e66d15e8e45727239cb08c9eeac26aa5befc4d2c: Status 404 returned error can't find the container with id 210463e3823a423c3f9d7804e66d15e8e45727239cb08c9eeac26aa5befc4d2c
Oct 30 19:37:36.383539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.383277    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:36.467720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.450865    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": context deadline exceeded"
Oct 30 19:37:36.543310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.506305    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:36.609814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.607804    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:37:36.609814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.607870    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:37:36.609814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.608287    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded"
Oct 30 19:37:36.609814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.609064    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:36.617187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.615798    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:36.617187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.616096    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:37:36.625470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.624765    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:36.724723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.724663    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:36.770722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.767875    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:36.812056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.812011    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:36.825753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.825716    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:37:36.895934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:36.895899    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:37:37.140257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.127533    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" podUID="aeff9d2a-d624-4ab3-a316-13d796122ee2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.187:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:37.140257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.138821    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:37:37.140257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.138898    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 19:37:37.202779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.202027    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:37.237473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.237425    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:37.279518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.279483    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:37:37.698715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.697989    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:37:37.698715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.698444    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:37:37.822736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.737220    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:37:37.822736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.737300    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 19:37:37.822736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.764118    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded"
Oct 30 19:37:37.822736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.764483    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": context deadline exceeded"
Oct 30 19:37:37.822736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.764773    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:37:37.861919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.848938    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:37.861919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.849379    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:37.869152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.869089    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:37.989849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.984943    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:37:37.990442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.990412    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:37.992736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:37.992711    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": context deadline exceeded"
Oct 30 19:37:38.004813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.004595    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:38.004813 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:37:38.004767    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550e4d4e_8cea_4f60_8061_2c5276c6e800.slice/crio-4c1595422483d06911cb8b4919e56c7e10ad921a295010ec93ebdef350d41855 WatchSource:0}: Error finding container 4c1595422483d06911cb8b4919e56c7e10ad921a295010ec93ebdef350d41855: Status 404 returned error can't find the container with id 4c1595422483d06911cb8b4919e56c7e10ad921a295010ec93ebdef350d41855
Oct 30 19:37:38.054718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.053983    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:38.083327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.082765    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:38.083934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.083899    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:37:38.084082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.083957    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 19:37:38.217773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.216876    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:38.272752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.266909    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:37:38.272752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.266973    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:37:38.272752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.267013    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:37:38.272752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.267052    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:37:38.272752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.267089    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:37:38.272752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.267119    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:37:38.272752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.267152    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:37:38.272752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.267185    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:37:38.272752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.267218    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:37:38.294538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.294503    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"]
Oct 30 19:37:38.507751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.507723    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:37:38.531319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.518543    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:37:38.531319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.518809    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:38.531319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.531140    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:37:38.551451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.551395    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" event={"ID":"cf6ec1b5-8f21-429e-a7fc-be52be068305","Type":"ContainerStarted","Data":"210463e3823a423c3f9d7804e66d15e8e45727239cb08c9eeac26aa5befc4d2c"}
Oct 30 19:37:38.592595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.592545    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:38.592895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.592866    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:38.593077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.593058    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:38.768418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.768357    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:37:38.769346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.769294    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:37:38.772974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.772837    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:37:38.790276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.788271    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": context deadline exceeded"
Oct 30 19:37:38.790815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.790798    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:37:38.791030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.791009    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:38.850292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.850251    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:38.890738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.883542    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:38.958738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.947841    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:38.959011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.958990    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:37:38.962704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.961394    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:37:38.962704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.961452    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:37:38.963590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.963571    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:38.977154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:38.977132    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:37:39.073254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:39.072084    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:37:39.073254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:39.072136    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Oct 30 19:37:39.282939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:39.246679    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:37:39.378027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:39.377914    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:37:39.465247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:39.465202    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podStartSLOduration=5.465186629 podStartE2EDuration="5.465186629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:39.36053589 +0000 UTC m=+1894.263548927" watchObservedRunningTime="2026-10-30 19:37:39.465186629 +0000 UTC m=+1894.368199727"
Oct 30 19:37:39.660918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:39.660882    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:39.937770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:39.911779    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:39.937770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:39.911848    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 19:37:39.971549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:39.952790    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" event={"ID":"56680a43-73b9-4d8f-865d-e112f76bfb19","Type":"ContainerStarted","Data":"72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d"}
Oct 30 19:37:40.091709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.091651    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" event={"ID":"550e4d4e-8cea-4f60-8061-2c5276c6e800","Type":"ContainerStarted","Data":"264ef4b3f41c2f8c52d00efd5175a6cc90a515e3dd365171d40e9d5444725275"}
Oct 30 19:37:40.092061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.092030    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" event={"ID":"550e4d4e-8cea-4f60-8061-2c5276c6e800","Type":"ContainerStarted","Data":"4c1595422483d06911cb8b4919e56c7e10ad921a295010ec93ebdef350d41855"}
Oct 30 19:37:40.129525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.129485    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" event={"ID":"cf6ec1b5-8f21-429e-a7fc-be52be068305","Type":"ContainerStarted","Data":"ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063"}
Oct 30 19:37:40.186730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.182904    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" podStartSLOduration=6.182884194 podStartE2EDuration="6.182884194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:40.173518575 +0000 UTC m=+1895.076531615" watchObservedRunningTime="2026-10-30 19:37:40.182884194 +0000 UTC m=+1895.085897233"
Oct 30 19:37:40.326770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.294182    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podStartSLOduration=5.294161677 podStartE2EDuration="5.294161677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:40.230105527 +0000 UTC m=+1895.133118565" watchObservedRunningTime="2026-10-30 19:37:40.294161677 +0000 UTC m=+1895.197174715"
Oct 30 19:37:40.351752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.320874    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:40.357224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.357164    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:40.376743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.375333    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:37:40.384027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.384001    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:40.385440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.385416    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:40.385649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.385619    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:37:40.475781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.475230    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:40.791673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.791644    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:40.793840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.793810    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:37:40.793984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.793865    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 19:37:40.868382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.868348    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:40.868560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:40.868401    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 19:37:41.126806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:41.125634    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:41.389785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:41.387874    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:41.389785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:41.387947    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:37:41.792523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:41.792427    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:37:41.792523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:41.792480    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 19:37:41.870723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:41.852263    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:41.874710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:41.871089    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:41.874710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:41.871150    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 19:37:42.386072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:42.386031    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:42.386236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:42.386096    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:37:42.565716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:42.560380    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:42.686991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:42.686961    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:42.790711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:42.786258    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:42.794716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:42.794311    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:37:42.794716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:42.794351    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 19:37:42.869050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:42.869014    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:42.869326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:42.869299    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 19:37:43.166786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:43.166750    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:43.418450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:43.418359    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:43.418730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:43.418702    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:37:43.683770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:43.682227    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:43.967671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:43.967580    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:37:44.027709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.002581    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:37:44.028013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.027972    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 19:37:44.051763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.050045    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:44.051763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.050100    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 19:37:44.075599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.075568    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:44.359046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.358937    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:44.372205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.363541    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.372205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.363977    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.409746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.405557    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:44.410064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.410014    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:37:44.494516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.494476    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.561244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.561204    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.586117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.586075    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": context deadline exceeded"
Oct 30 19:37:44.799713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.634971    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.826962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.635004    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded"
Oct 30 19:37:44.826962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.726538    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.826962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.758282    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:44.826962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.764205    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.826962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.813791    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:37:44.826962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.813842    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 19:37:44.870707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.868905    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:44.870707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.868988    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 19:37:44.935798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.935750    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded"
Oct 30 19:37:44.937571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.937537    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.946337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.946301    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.980260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.980223    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:37:44.982421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:44.982396    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:45.009714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.003444    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.009714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.005586    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.050339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.048291    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.070967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.070652    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.093613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.091793    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.180136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.176432    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:37:45.199833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.196809    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.390821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.389908    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:45.391004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.390947    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:45.391004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.390994    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:37:45.816858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.800360    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.816858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.801042    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:37:45.816858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.801484    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:37:45.816858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.801633    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 30 19:37:45.886275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.872422    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:45.886275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:45.872477    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 19:37:46.087216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.087119    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:46.178277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.113824    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.178277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.169464    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:37:46.248336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.248309    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:37:46.364763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.360517    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:37:46.386308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.385725    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:46.386308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.385779    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:37:46.781707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.781337    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:46.928844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.898720    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 19:37:46.928844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:46.898789    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 19:37:47.083941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:47.083829    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:47.085345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:47.085325    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:47.086332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:47.086311    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:37:47.384574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:47.384496    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:37:47.384574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:47.384553    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 19:37:47.684063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:47.684024    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:47.873050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:47.873022    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:47.873511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:47.873489    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:47.874198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:47.874179    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:37:48.388970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:48.388944    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:48.389421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:48.389238    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:48.390105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:48.390084    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:37:48.539656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:48.539617    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:49.365532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:49.364085    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:50.270624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:50.270586    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:51.090714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:51.090307    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:51.782239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:51.782192    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:52.084408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:52.084310    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:52.144055 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:37:52.144011    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:37:52.251054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:52.251020    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:52.373769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:52.372045    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"]
Oct 30 19:37:52.373769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:52.372267    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy" containerID="cri-o://f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134" gracePeriod=30
Oct 30 19:37:52.488211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:52.488169    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:52.984346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:52.984296    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:53.866215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:53.866184    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:54.570500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:54.570460    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:55.451709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.446067    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:55.744427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.744339    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"]
Oct 30 19:37:55.751917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.751895    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.762426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.762402    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"]
Oct 30 19:37:55.762615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.762571    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1b1531\"/\"openshift-service-ca.crt\""
Oct 30 19:37:55.762728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.762645    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-1b1531\"/\"kube-root-ca.crt\""
Oct 30 19:37:55.764101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.764082    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1b1531\"/\"default-dockercfg-2rpjp\""
Oct 30 19:37:55.768156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.768140    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-1b1531\"/\"auth-test-model-1b1531-kserve-self-signed-certs\""
Oct 30 19:37:55.788504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.788462    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed24353-f764-42b6-b674-628c07930cca-tls-certs\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.788720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.788650    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-model-cache\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.788969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.788946    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7cx\" (UniqueName: \"kubernetes.io/projected/5ed24353-f764-42b6-b674-628c07930cca-kube-api-access-lz7cx\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.789053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.789007    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-tmp-dir\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.789171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.789155    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-home\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.789252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.789233    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-dshm\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.860835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.860799    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:55.891383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.891345    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7cx\" (UniqueName: \"kubernetes.io/projected/5ed24353-f764-42b6-b674-628c07930cca-kube-api-access-lz7cx\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.891585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.891422    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-tmp-dir\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.891585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.891533    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-home\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.891796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.891613    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-dshm\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.891796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.891676    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed24353-f764-42b6-b674-628c07930cca-tls-certs\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.891925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.891814    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-model-cache\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.891986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.891934    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-tmp-dir\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.892067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.892046    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-home\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.892413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.892384    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-model-cache\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.894732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.894682    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-dshm\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.894822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.894800    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed24353-f764-42b6-b674-628c07930cca-tls-certs\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:55.902936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:55.902917    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7cx\" (UniqueName: \"kubernetes.io/projected/5ed24353-f764-42b6-b674-628c07930cca-kube-api-access-lz7cx\") pod \"auth-test-model-1b1531-kserve-c7698f8c7-vq789\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") " pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:56.096760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:56.087793    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:56.557931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:56.557890    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:56.612657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:56.612631    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"]
Oct 30 19:37:56.613283 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:37:56.613258    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ed24353_f764_42b6_b674_628c07930cca.slice/crio-8166a5d6e5c146d60a1e2be0b7c0a97b3e7d1dd6b602252ec730d211759131a8 WatchSource:0}: Error finding container 8166a5d6e5c146d60a1e2be0b7c0a97b3e7d1dd6b602252ec730d211759131a8: Status 404 returned error can't find the container with id 8166a5d6e5c146d60a1e2be0b7c0a97b3e7d1dd6b602252ec730d211759131a8
Oct 30 19:37:57.156762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.156682    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" event={"ID":"5ed24353-f764-42b6-b674-628c07930cca","Type":"ContainerStarted","Data":"c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418"}
Oct 30 19:37:57.157222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.156780    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" event={"ID":"5ed24353-f764-42b6-b674-628c07930cca","Type":"ContainerStarted","Data":"8166a5d6e5c146d60a1e2be0b7c0a97b3e7d1dd6b602252ec730d211759131a8"}
Oct 30 19:37:57.178037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.177991    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podStartSLOduration=2.177974852 podStartE2EDuration="2.177974852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:57.173868396 +0000 UTC m=+1912.076881435" watchObservedRunningTime="2026-10-30 19:37:57.177974852 +0000 UTC m=+1912.080987890"
Oct 30 19:37:57.230766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.230732    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:57.546179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.546147    4925 patch_prober.go:29] interesting pod/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 19:37:57.546395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.546194    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 19:37:57.700053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.700026    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:37:57.736212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.736176    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"]
Oct 30 19:37:57.737285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.737262    4925 cpu_manager.go:414] "containerMap: removing container" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy"
Oct 30 19:37:57.737285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.737284    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy"
Oct 30 19:37:57.737533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.737520    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerName="istio-proxy"
Oct 30 19:37:57.741990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.741967    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.755776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.754802    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"]
Oct 30 19:37:57.760739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.760670    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:57.766404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.766376    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-envoy\") pod \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") "
Oct 30 19:37:57.766624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.766517    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-token\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-token\") pod \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") "
Oct 30 19:37:57.766624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.766611    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-podinfo\") pod \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") "
Oct 30 19:37:57.766836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.766661    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istiod-ca-cert\") pod \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") "
Oct 30 19:37:57.766836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.766713    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-certs\") pod \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") "
Oct 30 19:37:57.767398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.767367    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istiod-ca-cert" pod "304ce70e-b48b-4ab2-bdc7-ee6206e3143d" (UID: "304ce70e-b48b-4ab2-bdc7-ee6206e3143d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:57.767524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.767502    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-certs" pod "304ce70e-b48b-4ab2-bdc7-ee6206e3143d" (UID: "304ce70e-b48b-4ab2-bdc7-ee6206e3143d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:57.767624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.767604    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-data\") pod \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") "
Oct 30 19:37:57.767841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.767810    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-kube-api-access-wwf8v\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-kube-api-access-wwf8v\") pod \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") "
Oct 30 19:37:57.767931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.767908    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-data" pod "304ce70e-b48b-4ab2-bdc7-ee6206e3143d" (UID: "304ce70e-b48b-4ab2-bdc7-ee6206e3143d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:57.768003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.767984    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-credential-socket\") pod \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") "
Oct 30 19:37:57.768071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.768051    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-socket\") pod \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\" (UID: \"304ce70e-b48b-4ab2-bdc7-ee6206e3143d\") "
Oct 30 19:37:57.768303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.768283    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-credential-socket" pod "304ce70e-b48b-4ab2-bdc7-ee6206e3143d" (UID: "304ce70e-b48b-4ab2-bdc7-ee6206e3143d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:57.768484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.768464    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-socket" pod "304ce70e-b48b-4ab2-bdc7-ee6206e3143d" (UID: "304ce70e-b48b-4ab2-bdc7-ee6206e3143d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:57.769215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.769176    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.769215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.769202    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.769215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.769216    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.769479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.769230    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.769479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.769242    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.773455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.773433    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-token" pod "304ce70e-b48b-4ab2-bdc7-ee6206e3143d" (UID: "304ce70e-b48b-4ab2-bdc7-ee6206e3143d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:57.774193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.774161    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-podinfo" pod "304ce70e-b48b-4ab2-bdc7-ee6206e3143d" (UID: "304ce70e-b48b-4ab2-bdc7-ee6206e3143d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:57.774806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.774577    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-kube-api-access-wwf8v" pod "304ce70e-b48b-4ab2-bdc7-ee6206e3143d" (UID: "304ce70e-b48b-4ab2-bdc7-ee6206e3143d"). InnerVolumeSpecName "kube-api-access-wwf8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:57.776169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.776145    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-envoy" pod "304ce70e-b48b-4ab2-bdc7-ee6206e3143d" (UID: "304ce70e-b48b-4ab2-bdc7-ee6206e3143d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:57.871329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.871298    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.871501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.871424    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-certs\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.871501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.871493    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-envoy\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.871635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.871621    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-podinfo\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.871673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.871663    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-data\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.871741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.871729    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-token\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.871822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.871804    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-credential-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.871919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.871906    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780f48c3-4607-44e3-88b4-4f91728ebe0d-istiod-ca-cert\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.871984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.871972    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xcg\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-kube-api-access-w7xcg\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.872126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.872116    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.872161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.872131    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwf8v\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-kube-api-access-wwf8v\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.872161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.872140    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.872161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.872149    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304ce70e-b48b-4ab2-bdc7-ee6206e3143d-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.973253 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973221    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-podinfo\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973276    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-data\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973332    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-token\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973362    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-credential-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973440    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780f48c3-4607-44e3-88b4-4f91728ebe0d-istiod-ca-cert\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973524    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xcg\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-kube-api-access-w7xcg\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973574    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973624    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-data\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973652    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-certs\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973750    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-envoy\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.973930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973812    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-credential-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.974044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.973994    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-certs\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.974044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.974018    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-socket\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.974509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.974476    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780f48c3-4607-44e3-88b4-4f91728ebe0d-istiod-ca-cert\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.976039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.976016    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-envoy\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.976131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.976089    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-podinfo\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.985457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.985430    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xcg\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-kube-api-access-w7xcg\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:57.985615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:57.985559    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-token\") pod \"e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") " pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:58.069585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.069513    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:37:58.088678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.088650    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:37:58.132713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.131951    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:58.185613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.185109    4925 generic.go:356] "Generic (PLEG): container finished" podID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" containerID="f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134" exitCode=0
Oct 30 19:37:58.188278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.186269    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"
Oct 30 19:37:58.202811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.201952    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" event={"ID":"304ce70e-b48b-4ab2-bdc7-ee6206e3143d","Type":"ContainerDied","Data":"f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134"}
Oct 30 19:37:58.202811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.202033    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x" event={"ID":"304ce70e-b48b-4ab2-bdc7-ee6206e3143d","Type":"ContainerDied","Data":"b68003503461c02cf3eea07432010732e0a94b1fdaa7ea3af7e5cc49c51b405a"}
Oct 30 19:37:58.202811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.202067    4925 scope.go:122] "RemoveContainer" containerID="f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134"
Oct 30 19:37:58.229951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.220667    4925 scope.go:122] "RemoveContainer" containerID="f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134"
Oct 30 19:37:58.236262 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:37:58.236227    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134\": container with ID starting with f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134 not found: ID does not exist" containerID="f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134"
Oct 30 19:37:58.236407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.236276    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134"} err="failed to get container status \"f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134\": rpc error: code = NotFound desc = could not find container \"f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134\": container with ID starting with f19e3328e631355c4c195ee39359f3891b49b5831abc30b20700325aebb7f134 not found: ID does not exist"
Oct 30 19:37:58.260236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.260208    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"]
Oct 30 19:37:58.270379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.270354    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c24c01-openshift-default-7b7464ff5c-fbc4x"]
Oct 30 19:37:58.414065 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:37:58.414038    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780f48c3_4607_44e3_88b4_4f91728ebe0d.slice/crio-596b78ac4badf9f361e3d81a93eddc8ba189cfba5722985db9853fa6927072d6 WatchSource:0}: Error finding container 596b78ac4badf9f361e3d81a93eddc8ba189cfba5722985db9853fa6927072d6: Status 404 returned error can't find the container with id 596b78ac4badf9f361e3d81a93eddc8ba189cfba5722985db9853fa6927072d6
Oct 30 19:37:58.414180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.414166    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"]
Oct 30 19:37:58.417292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.417259    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:58.417436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.417333    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:58.417436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.417378    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:58.853958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:58.851350    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:59.211715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:59.196276    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" event={"ID":"780f48c3-4607-44e3-88b4-4f91728ebe0d","Type":"ContainerStarted","Data":"93aa2b3683bbd2b49a4b20158c63a5daaed6519df9bfd7f7e44e6e0c84b27a32"}
Oct 30 19:37:59.211715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:59.196373    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" event={"ID":"780f48c3-4607-44e3-88b4-4f91728ebe0d","Type":"ContainerStarted","Data":"596b78ac4badf9f361e3d81a93eddc8ba189cfba5722985db9853fa6927072d6"}
Oct 30 19:37:59.223586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:59.223524    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" podStartSLOduration=4.223504502 podStartE2EDuration="4.223504502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:59.221439625 +0000 UTC m=+1914.124452664" watchObservedRunningTime="2026-10-30 19:37:59.223504502 +0000 UTC m=+1914.126517541"
Oct 30 19:37:59.265849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:59.265818    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304ce70e-b48b-4ab2-bdc7-ee6206e3143d" path="/var/lib/kubelet/pods/304ce70e-b48b-4ab2-bdc7-ee6206e3143d/volumes"
Oct 30 19:37:59.576773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:37:59.573882    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:00.070045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:00.070010    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:38:00.071438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:00.071413    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:38:00.071583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:00.071461    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 19:38:00.435725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:00.435323    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:01.069894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:01.069859    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:38:01.070058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:01.069920    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 19:38:01.171015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:01.170400    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:01.888725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:01.881307    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:02.070117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:02.070080    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:38:02.070282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:02.070143    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 19:38:02.678500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:02.678466    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:03.070108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:03.070025    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:38:03.070593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:03.070566    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 19:38:03.237283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:03.237249    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:03.515805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:03.490789    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:04.070271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:04.070226    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:38:04.070798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:04.070310    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 19:38:04.210716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:04.209854    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:04.927766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:04.927733    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:05.070599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:05.070557    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 19:38:05.071056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:05.070615    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 19:38:05.784714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:05.784318    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:06.076482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:06.076395    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:38:06.076960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:06.076942    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:38:06.077668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:06.077643    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:38:06.089040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:06.089022    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:38:06.101865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:06.101842    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:38:06.309554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:06.309525    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:38:06.627912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:06.627875    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:07.271320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:07.268528    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:07.949735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:07.948656    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:08.704745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:08.704714    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:09.447710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:09.447564    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:10.061335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:10.061294    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:10.682726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:10.681154    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:11.120990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:11.120903    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"]
Oct 30 19:38:11.122600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:11.122493    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg" podUID="6cbdf529-2c4d-4fbb-a526-4020fb2cab17" containerName="maas-api" containerID="cri-o://7359c64e28a4c4351e989a41c24469551e703a98a0c6d7497008613c89715563" gracePeriod=30
Oct 30 19:38:11.229614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:11.229573    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:11.416720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:11.411191    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:11.756811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:11.640982    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:11.868809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:11.851957    4925 generic.go:356] "Generic (PLEG): container finished" podID="6cbdf529-2c4d-4fbb-a526-4020fb2cab17" containerID="7359c64e28a4c4351e989a41c24469551e703a98a0c6d7497008613c89715563" exitCode=0
Oct 30 19:38:11.935184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:11.852066    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg" event={"ID":"6cbdf529-2c4d-4fbb-a526-4020fb2cab17","Type":"ContainerDied","Data":"7359c64e28a4c4351e989a41c24469551e703a98a0c6d7497008613c89715563"}
Oct 30 19:38:11.947662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:11.943362    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:12.013007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.012943    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:38:12.043431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.043402    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-kube-api-access-bbmpk\" (UniqueName: \"kubernetes.io/projected/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-kube-api-access-bbmpk\") pod \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") "
Oct 30 19:38:12.043586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.043454    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-metrics-certs\") pod \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") "
Oct 30 19:38:12.043656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.043634    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-maas-api-tls\") pod \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") "
Oct 30 19:38:12.043770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.043749    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-openshift-service-ca\") pod \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\" (UID: \"6cbdf529-2c4d-4fbb-a526-4020fb2cab17\") "
Oct 30 19:38:12.044163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.044104    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-openshift-service-ca" pod "6cbdf529-2c4d-4fbb-a526-4020fb2cab17" (UID: "6cbdf529-2c4d-4fbb-a526-4020fb2cab17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:12.044951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.044929    4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-openshift-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:12.045744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.045716    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-metrics-certs" pod "6cbdf529-2c4d-4fbb-a526-4020fb2cab17" (UID: "6cbdf529-2c4d-4fbb-a526-4020fb2cab17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:12.045950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.045932    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-kube-api-access-bbmpk" pod "6cbdf529-2c4d-4fbb-a526-4020fb2cab17" (UID: "6cbdf529-2c4d-4fbb-a526-4020fb2cab17"). InnerVolumeSpecName "kube-api-access-bbmpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:12.046019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.045986    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-maas-api-tls" pod "6cbdf529-2c4d-4fbb-a526-4020fb2cab17" (UID: "6cbdf529-2c4d-4fbb-a526-4020fb2cab17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:12.097018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.096972    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"]
Oct 30 19:38:12.097836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.097819    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6cbdf529-2c4d-4fbb-a526-4020fb2cab17" containerName="maas-api"
Oct 30 19:38:12.097888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.097838    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cbdf529-2c4d-4fbb-a526-4020fb2cab17" containerName="maas-api"
Oct 30 19:38:12.097975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.097966    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cbdf529-2c4d-4fbb-a526-4020fb2cab17" containerName="maas-api"
Oct 30 19:38:12.101981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.101958    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.139074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.139049    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-1b1531-openshift-default-dockercfg-5ppz9\""
Oct 30 19:38:12.146311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.146285    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"]
Oct 30 19:38:12.146841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.146818    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.146926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.146863    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.146926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.146889    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmf54\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-kube-api-access-fmf54\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.147048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.147031    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.147174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.147161    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.147244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.147230    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fd71d3-52ae-490a-96a2-3676b5a508bc-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.147277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.147269    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.147313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.147299    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.147346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.147316    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.147555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.147543    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-maas-api-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:12.147600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.147561    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbmpk\" (UniqueName: \"kubernetes.io/projected/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-kube-api-access-bbmpk\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:12.147600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.147571    4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cbdf529-2c4d-4fbb-a526-4020fb2cab17-metrics-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:12.249095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249059    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fd71d3-52ae-490a-96a2-3676b5a508bc-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249112    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249160    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249179    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249426    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249475    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249509    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmf54\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-kube-api-access-fmf54\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249580    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249589    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249635    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249666    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249814    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249903    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.249971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.249930    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fd71d3-52ae-490a-96a2-3676b5a508bc-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.252207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.252183    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.252442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.252422    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.272326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.272267    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmf54\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-kube-api-access-fmf54\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.272925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.272903    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.370090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.370048    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:12.417789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.417757    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:12.723710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.723587    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"]
Oct 30 19:38:12.724348 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:38:12.724321    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fd71d3_52ae_490a_96a2_3676b5a508bc.slice/crio-4ee3de7bb9869b756eece8fc837730e8154a0f86a933074508365a22c4c88153 WatchSource:0}: Error finding container 4ee3de7bb9869b756eece8fc837730e8154a0f86a933074508365a22c4c88153: Status 404 returned error can't find the container with id 4ee3de7bb9869b756eece8fc837730e8154a0f86a933074508365a22c4c88153
Oct 30 19:38:12.729211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.729174    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:12.729305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.729250    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:12.729305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.729289    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:12.869573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.869534    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg" event={"ID":"6cbdf529-2c4d-4fbb-a526-4020fb2cab17","Type":"ContainerDied","Data":"7d0a97c3df14483a9ded29fd2b8a4446ce9a02cf7fd8c2c2cd0ac121a001414b"}
Oct 30 19:38:12.869830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.869602    4925 scope.go:122] "RemoveContainer" containerID="7359c64e28a4c4351e989a41c24469551e703a98a0c6d7497008613c89715563"
Oct 30 19:38:12.870431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.869936    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"
Oct 30 19:38:12.873929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.873900    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" event={"ID":"95fd71d3-52ae-490a-96a2-3676b5a508bc","Type":"ContainerStarted","Data":"47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0"}
Oct 30 19:38:12.874060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.873955    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" event={"ID":"95fd71d3-52ae-490a-96a2-3676b5a508bc","Type":"ContainerStarted","Data":"4ee3de7bb9869b756eece8fc837730e8154a0f86a933074508365a22c4c88153"}
Oct 30 19:38:12.921982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.921828    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:12.945573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.945497    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" podStartSLOduration=12.945474021999999 podStartE2EDuration="12.945474022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:12.908768822 +0000 UTC m=+1927.811781861" watchObservedRunningTime="2026-10-30 19:38:12.945474022 +0000 UTC m=+1927.848487060"
Oct 30 19:38:12.986826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.983968    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"]
Oct 30 19:38:12.988462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:12.988437    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg"]
Oct 30 19:38:13.269811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:13.269725    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cbdf529-2c4d-4fbb-a526-4020fb2cab17" path="/var/lib/kubelet/pods/6cbdf529-2c4d-4fbb-a526-4020fb2cab17/volumes"
Oct 30 19:38:13.418793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:13.418760    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:13.420229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:13.420198    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:38:13.420353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:13.420256    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:38:13.616767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:13.612656    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:14.228713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:14.219129    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:14.419714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:14.419487    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:38:14.419714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:14.419556    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:38:14.456794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:14.456752    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-76117c-5c8dbbff75-pmfwg" podUID="6cbdf529-2c4d-4fbb-a526-4020fb2cab17" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:15.004130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:15.004097    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:15.420713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:15.418756    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:38:15.420713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:15.418816    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:38:15.658742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:15.658062    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:16.432863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:16.432770    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:38:16.433468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:16.433425    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:38:16.468715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:16.465716    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:17.373309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:17.373266    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:17.425144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:17.425114    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:38:17.425375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:17.425354    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:38:18.281742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:18.281602    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:18.423535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:18.422992    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:38:18.423535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:18.423070    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:38:18.948209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:18.948156    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:19.422937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.422910    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:19.423404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.423329    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:19.424069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.424049    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:38:19.467784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.467749    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:19.699358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.699261    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:19.768465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.768429    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"]
Oct 30 19:38:19.768878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.768845    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy" containerID="cri-o://93aa2b3683bbd2b49a4b20158c63a5daaed6519df9bfd7f7e44e6e0c84b27a32" gracePeriod=30
Oct 30 19:38:19.776094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.776072    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"]
Oct 30 19:38:19.776423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.776320    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy" containerID="cri-o://996e1b75021b6ff20d8e7424387faf473663e92dceab7f578f35e298e1b10203" gracePeriod=30
Oct 30 19:38:19.935959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:19.935789    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:20.295434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:20.295400    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:38:20.295610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:20.295462    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:38:20.595084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:20.594983    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:20.760805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:20.760773    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:21.079609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:21.079579    4925 patch_prober.go:29] interesting pod/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:38:21.079809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:21.079643    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:38:21.299715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:21.283333    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:21.509985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:21.509953    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:21.832769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:21.832209    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:22.127188 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:38:22.127091    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:38:23.513810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.513764    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.539463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.539418    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.659663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.659624    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.687809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.687767    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:38:23.688426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.688396    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/health\": context deadline exceeded"
Oct 30 19:38:23.742894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.742852    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.802897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.802786    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.852048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.852000    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.876848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.876797    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": context deadline exceeded"
Oct 30 19:38:23.892394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.892362    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:23.910875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.910820    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:23.966845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:23.966795    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.015719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.002771    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.078441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.078338    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" podUID="aeff9d2a-d624-4ab3-a316-13d796122ee2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.187:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.181721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.181652    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.260843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.260793    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.272014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.271978    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.338968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.338871    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.356158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.356116    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.369026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.368986    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.447862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.445787    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.493009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.491886    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.527718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.518777    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.527718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.519238    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:38:24.553711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.550447    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.553711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.551112    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:38:24.629779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.582932    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.690896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.690826    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.691820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.691793    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:38:24.801713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.768843    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.801713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.769541    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:38:24.803420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.803378    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:24.804203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.804180    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:38:24.976984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.912326    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.050128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.972772    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.051329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.978069    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.098817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.082566    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:38:25.098817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.082648    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:38:25.098817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.989862    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:25.109990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.109824    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" podUID="aeff9d2a-d624-4ab3-a316-13d796122ee2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.187:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.110166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.110129    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/health\": context deadline exceeded"
Oct 30 19:38:25.110269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.003709    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.110412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.110390    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:38:25.110486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.110454    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:38:25.110948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:24.978133    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.116821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.116798    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:38:25.121920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.003212    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.175817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.000974    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.175817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.142635    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:38:25.387840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.382809    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.419975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.416508    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.419975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.417281    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Oct 30 19:38:25.422071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.420924    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.426743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.422289    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.547667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.547629    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:38:25.554941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.554897    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.627291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.604054    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.627291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.610518    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:38:25.699561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.699023    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.699561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.699108    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.774124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.771780    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:25.857466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.852162    4925 generic.go:356] "Generic (PLEG): container finished" podID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerID="996e1b75021b6ff20d8e7424387faf473663e92dceab7f578f35e298e1b10203" exitCode=0
Oct 30 19:38:25.862718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.862235    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:25.911185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.909891    4925 generic.go:356] "Generic (PLEG): container finished" podID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerID="93aa2b3683bbd2b49a4b20158c63a5daaed6519df9bfd7f7e44e6e0c84b27a32" exitCode=0
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.936850    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.936914    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.936932    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" event={"ID":"63c4a300-1b96-4a15-a0c2-cd8ca681129f","Type":"ContainerDied","Data":"996e1b75021b6ff20d8e7424387faf473663e92dceab7f578f35e298e1b10203"}
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.936991    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937039    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937072    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937106    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937134    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937166    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937180    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" event={"ID":"780f48c3-4607-44e3-88b4-4f91728ebe0d","Type":"ContainerDied","Data":"93aa2b3683bbd2b49a4b20158c63a5daaed6519df9bfd7f7e44e6e0c84b27a32"}
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937225    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937258    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937287    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937318    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937352    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937383    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937421    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937450    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937486    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937513    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937560    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937591    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:38:25.939156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.937620    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:38:25.940564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:25.939530    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:38:25.979908 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:38:25.979873    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c4a300_1b96_4a15_a0c2_cd8ca681129f.slice/crio-conmon-996e1b75021b6ff20d8e7424387faf473663e92dceab7f578f35e298e1b10203.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:38:26.163047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.163015    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:38:26.265859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.265510    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-envoy\") pod \"780f48c3-4607-44e3-88b4-4f91728ebe0d\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") "
Oct 30 19:38:26.265859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.265552    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-socket\") pod \"780f48c3-4607-44e3-88b4-4f91728ebe0d\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") "
Oct 30 19:38:26.265859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.265576    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-podinfo\") pod \"780f48c3-4607-44e3-88b4-4f91728ebe0d\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") "
Oct 30 19:38:26.265859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.265624    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780f48c3-4607-44e3-88b4-4f91728ebe0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780f48c3-4607-44e3-88b4-4f91728ebe0d-istiod-ca-cert\") pod \"780f48c3-4607-44e3-88b4-4f91728ebe0d\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") "
Oct 30 19:38:26.265859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.265682    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-credential-socket\") pod \"780f48c3-4607-44e3-88b4-4f91728ebe0d\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") "
Oct 30 19:38:26.265859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.265752    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-certs\") pod \"780f48c3-4607-44e3-88b4-4f91728ebe0d\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") "
Oct 30 19:38:26.266368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.265987    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-kube-api-access-w7xcg\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-kube-api-access-w7xcg\") pod \"780f48c3-4607-44e3-88b4-4f91728ebe0d\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") "
Oct 30 19:38:26.266368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.266271    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-data\") pod \"780f48c3-4607-44e3-88b4-4f91728ebe0d\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") "
Oct 30 19:38:26.266368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.266319    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-token\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-token\") pod \"780f48c3-4607-44e3-88b4-4f91728ebe0d\" (UID: \"780f48c3-4607-44e3-88b4-4f91728ebe0d\") "
Oct 30 19:38:26.280554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.271586    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-token" pod "780f48c3-4607-44e3-88b4-4f91728ebe0d" (UID: "780f48c3-4607-44e3-88b4-4f91728ebe0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:26.280554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.271823    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-credential-socket" pod "780f48c3-4607-44e3-88b4-4f91728ebe0d" (UID: "780f48c3-4607-44e3-88b4-4f91728ebe0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.280554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.271996    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-certs" pod "780f48c3-4607-44e3-88b4-4f91728ebe0d" (UID: "780f48c3-4607-44e3-88b4-4f91728ebe0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.280554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.272517    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-socket" pod "780f48c3-4607-44e3-88b4-4f91728ebe0d" (UID: "780f48c3-4607-44e3-88b4-4f91728ebe0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.280554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.273060    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-data" pod "780f48c3-4607-44e3-88b4-4f91728ebe0d" (UID: "780f48c3-4607-44e3-88b4-4f91728ebe0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.280554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.279841    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-envoy" pod "780f48c3-4607-44e3-88b4-4f91728ebe0d" (UID: "780f48c3-4607-44e3-88b4-4f91728ebe0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.281748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.281723    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780f48c3-4607-44e3-88b4-4f91728ebe0d-istiod-ca-cert" pod "780f48c3-4607-44e3-88b4-4f91728ebe0d" (UID: "780f48c3-4607-44e3-88b4-4f91728ebe0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:26.284576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.284548    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-podinfo" pod "780f48c3-4607-44e3-88b4-4f91728ebe0d" (UID: "780f48c3-4607-44e3-88b4-4f91728ebe0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:26.286471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.286445    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-kube-api-access-w7xcg" pod "780f48c3-4607-44e3-88b4-4f91728ebe0d" (UID: "780f48c3-4607-44e3-88b4-4f91728ebe0d"). InnerVolumeSpecName "kube-api-access-w7xcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:26.304792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.304642    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:38:26.368838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.368799    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-certs\") pod \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") "
Oct 30 19:38:26.369029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.368908    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istiod-ca-cert\") pod \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") "
Oct 30 19:38:26.369029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.368979    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-kube-api-access-7cpkd\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-kube-api-access-7cpkd\") pod \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") "
Oct 30 19:38:26.369223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.369063    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-credential-socket\") pod \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") "
Oct 30 19:38:26.369223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.369098    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-data\") pod \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") "
Oct 30 19:38:26.369384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.369363    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-envoy\") pod \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") "
Oct 30 19:38:26.369460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.369443    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-podinfo\") pod \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") "
Oct 30 19:38:26.369589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.369547    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-socket\") pod \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") "
Oct 30 19:38:26.369669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.369655    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-token\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-token\") pod \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\" (UID: \"63c4a300-1b96-4a15-a0c2-cd8ca681129f\") "
Oct 30 19:38:26.370620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.370602    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.370620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.370622    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.370802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.370641    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.370802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.370651    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.370802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.370659    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780f48c3-4607-44e3-88b4-4f91728ebe0d-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.370802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.370667    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.370802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.370675    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.370802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.370705    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xcg\" (UniqueName: \"kubernetes.io/projected/780f48c3-4607-44e3-88b4-4f91728ebe0d-kube-api-access-w7xcg\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.370802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.370717    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780f48c3-4607-44e3-88b4-4f91728ebe0d-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.371172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.371018    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-data" pod "63c4a300-1b96-4a15-a0c2-cd8ca681129f" (UID: "63c4a300-1b96-4a15-a0c2-cd8ca681129f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.371172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.371035    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-socket" pod "63c4a300-1b96-4a15-a0c2-cd8ca681129f" (UID: "63c4a300-1b96-4a15-a0c2-cd8ca681129f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.371294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.371246    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-certs" pod "63c4a300-1b96-4a15-a0c2-cd8ca681129f" (UID: "63c4a300-1b96-4a15-a0c2-cd8ca681129f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.371294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.371273    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istiod-ca-cert" pod "63c4a300-1b96-4a15-a0c2-cd8ca681129f" (UID: "63c4a300-1b96-4a15-a0c2-cd8ca681129f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:26.371538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.371508    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-credential-socket" pod "63c4a300-1b96-4a15-a0c2-cd8ca681129f" (UID: "63c4a300-1b96-4a15-a0c2-cd8ca681129f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.374659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.374102    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-envoy" pod "63c4a300-1b96-4a15-a0c2-cd8ca681129f" (UID: "63c4a300-1b96-4a15-a0c2-cd8ca681129f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:26.374659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.374530    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-kube-api-access-7cpkd" pod "63c4a300-1b96-4a15-a0c2-cd8ca681129f" (UID: "63c4a300-1b96-4a15-a0c2-cd8ca681129f"). InnerVolumeSpecName "kube-api-access-7cpkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:26.375270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.375241    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-token" pod "63c4a300-1b96-4a15-a0c2-cd8ca681129f" (UID: "63c4a300-1b96-4a15-a0c2-cd8ca681129f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:26.375270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.375248    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-podinfo" pod "63c4a300-1b96-4a15-a0c2-cd8ca681129f" (UID: "63c4a300-1b96-4a15-a0c2-cd8ca681129f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:26.500796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.495120    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.500796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.495155    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.500796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.495172    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.500796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.495192    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.500796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.495205    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.500796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.495217    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.500796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.495231    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cpkd\" (UniqueName: \"kubernetes.io/projected/63c4a300-1b96-4a15-a0c2-cd8ca681129f-kube-api-access-7cpkd\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.500796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.495253    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.500796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.495265    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c4a300-1b96-4a15-a0c2-cd8ca681129f-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:26.536674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.536639    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"]
Oct 30 19:38:26.537794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.537769    4925 cpu_manager.go:414] "containerMap: removing container" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy"
Oct 30 19:38:26.537952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.537939    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy"
Oct 30 19:38:26.538105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.538095    4925 cpu_manager.go:414] "containerMap: removing container" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy"
Oct 30 19:38:26.538175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.538165    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy"
Oct 30 19:38:26.538447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.538434    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" containerName="istio-proxy"
Oct 30 19:38:26.538550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.538539    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" containerName="istio-proxy"
Oct 30 19:38:26.545579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.545546    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"]
Oct 30 19:38:26.545738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.545663    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.565427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.565400    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"]
Oct 30 19:38:26.571875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.571845    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"]
Oct 30 19:38:26.571981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.571947    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.699464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.699425    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.699661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.699572    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.699661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.699611    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.699661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.699648    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.699826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.699781    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.699910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.699884    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.699989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.699948    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.700046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700029    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxd47\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-kube-api-access-qxd47\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.700127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700112    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf1b65c-2559-4d34-9539-448e37119221-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.700209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700194    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.700257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700230    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf1b65c-2559-4d34-9539-448e37119221-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.700296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700269    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.700409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700383    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.700508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700475    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137fbb43-62a3-415b-9cd9-15dad618d603-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.700577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700531    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.700577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700557    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137fbb43-62a3-415b-9cd9-15dad618d603-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.700733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700582    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.700733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.700641    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6s6s\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-kube-api-access-j6s6s\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.803030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.802951    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxd47\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-kube-api-access-qxd47\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.803194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803027    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf1b65c-2559-4d34-9539-448e37119221-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.803194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803063    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.803194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803103    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf1b65c-2559-4d34-9539-448e37119221-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.803194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803141    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.803194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803192    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.803478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803264    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137fbb43-62a3-415b-9cd9-15dad618d603-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.803478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803315    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.803478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803339    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137fbb43-62a3-415b-9cd9-15dad618d603-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.803478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803362    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.803478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803415    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6s6s\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-kube-api-access-j6s6s\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.803825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803535    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.803825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803575    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.803825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803668    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.803825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803723    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.803825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803754    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.803825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803764    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.804148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803878    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.804148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803939    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.804148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.803991    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.804148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.804125    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf1b65c-2559-4d34-9539-448e37119221-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.804406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.804364    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.804482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.804449    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-data\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.804627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.804608    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.804740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.804633    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137fbb43-62a3-415b-9cd9-15dad618d603-istiod-ca-cert\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.804931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.804908    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.805008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.804955    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-credential-socket\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.805245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.805220    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-certs\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.806183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.806161    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf1b65c-2559-4d34-9539-448e37119221-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.806321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.806301    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.806509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.806493    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-envoy\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.807369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.807351    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137fbb43-62a3-415b-9cd9-15dad618d603-istio-podinfo\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.814971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.814943    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxd47\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-kube-api-access-qxd47\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.815851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.815829    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.815929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.815893    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-istio-token\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.816724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.816707    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6s6s\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-kube-api-access-j6s6s\") pod \"e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") " pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:26.865778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.865746    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:26.897624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:26.897595    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:27.027781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.026650    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"
Oct 30 19:38:27.027781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.027122    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8879-af7d4e"]
Oct 30 19:38:27.041797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.040407    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"
Oct 30 19:38:27.064221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.064145    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8879-af7d4e"]
Oct 30 19:38:27.064477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.064456    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4" event={"ID":"63c4a300-1b96-4a15-a0c2-cd8ca681129f","Type":"ContainerDied","Data":"3acc5a92d8d99d2a1879624dd9eb9c3cc09222a1cef5784593b4f568351183e9"}
Oct 30 19:38:27.064990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.064964    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8" event={"ID":"780f48c3-4607-44e3-88b4-4f91728ebe0d","Type":"ContainerDied","Data":"596b78ac4badf9f361e3d81a93eddc8ba189cfba5722985db9853fa6927072d6"}
Oct 30 19:38:27.065170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.065156    4925 scope.go:122] "RemoveContainer" containerID="996e1b75021b6ff20d8e7424387faf473663e92dceab7f578f35e298e1b10203"
Oct 30 19:38:27.065491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.065475    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:27.074301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.073268    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bmmlc\""
Oct 30 19:38:27.074301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.074219    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:27.138718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.138216    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcd2b\" (UniqueName: \"kubernetes.io/projected/bcbe8125-f207-441b-bb1f-2e776302c468-kube-api-access-tcd2b\") pod \"test-curl-8879-af7d4e\" (UID: \"bcbe8125-f207-441b-bb1f-2e776302c468\") " pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:27.198358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.198333    4925 scope.go:122] "RemoveContainer" containerID="93aa2b3683bbd2b49a4b20158c63a5daaed6519df9bfd7f7e44e6e0c84b27a32"
Oct 30 19:38:27.209367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.209340    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"]
Oct 30 19:38:27.221618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.221591    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-rk8c8"]
Oct 30 19:38:27.229840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.229815    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"]
Oct 30 19:38:27.239436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.239391    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-76117c-openshift-default-5bf558ddbc-c8tp4"]
Oct 30 19:38:27.240092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.240058    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcd2b\" (UniqueName: \"kubernetes.io/projected/bcbe8125-f207-441b-bb1f-2e776302c468-kube-api-access-tcd2b\") pod \"test-curl-8879-af7d4e\" (UID: \"bcbe8125-f207-441b-bb1f-2e776302c468\") " pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:27.277795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.277765    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcd2b\" (UniqueName: \"kubernetes.io/projected/bcbe8125-f207-441b-bb1f-2e776302c468-kube-api-access-tcd2b\") pod \"test-curl-8879-af7d4e\" (UID: \"bcbe8125-f207-441b-bb1f-2e776302c468\") " pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:27.287183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.287155    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c4a300-1b96-4a15-a0c2-cd8ca681129f" path="/var/lib/kubelet/pods/63c4a300-1b96-4a15-a0c2-cd8ca681129f/volumes"
Oct 30 19:38:27.287854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.287835    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780f48c3-4607-44e3-88b4-4f91728ebe0d" path="/var/lib/kubelet/pods/780f48c3-4607-44e3-88b4-4f91728ebe0d/volumes"
Oct 30 19:38:27.331847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.331818    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"]
Oct 30 19:38:27.331847 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:38:27.331833    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf1b65c_2559_4d34_9539_448e37119221.slice/crio-6140406f00c50b66e70a70fbffbd4aee78aaac1cae5181edac03702c19863200 WatchSource:0}: Error finding container 6140406f00c50b66e70a70fbffbd4aee78aaac1cae5181edac03702c19863200: Status 404 returned error can't find the container with id 6140406f00c50b66e70a70fbffbd4aee78aaac1cae5181edac03702c19863200
Oct 30 19:38:27.335173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.335137    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:27.335295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.335204    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:27.335295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.335235    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:27.349651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.349627    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"]
Oct 30 19:38:27.353290 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:38:27.353258    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137fbb43_62a3_415b_9cd9_15dad618d603.slice/crio-4309beefbee1af7cbb0ae997082c908df7478329412a83c37416b05914690b06 WatchSource:0}: Error finding container 4309beefbee1af7cbb0ae997082c908df7478329412a83c37416b05914690b06: Status 404 returned error can't find the container with id 4309beefbee1af7cbb0ae997082c908df7478329412a83c37416b05914690b06
Oct 30 19:38:27.356574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.356536    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:27.356801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.356775    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:27.356938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.356919    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:38:27.495155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.494915    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:27.791388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.789626    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:27.831739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:27.831594    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8879-af7d4e"]
Oct 30 19:38:27.832601 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:38:27.832569    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcbe8125_f207_441b_bb1f_2e776302c468.slice/crio-a2cc0eabc0d71a4dba1dbc58a5eaa9a8fa416c0b373f42f9a7ee847c57ba1558 WatchSource:0}: Error finding container a2cc0eabc0d71a4dba1dbc58a5eaa9a8fa416c0b373f42f9a7ee847c57ba1558: Status 404 returned error can't find the container with id a2cc0eabc0d71a4dba1dbc58a5eaa9a8fa416c0b373f42f9a7ee847c57ba1558
Oct 30 19:38:28.051835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.051802    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" event={"ID":"6cf1b65c-2559-4d34-9539-448e37119221","Type":"ContainerStarted","Data":"e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417"}
Oct 30 19:38:28.052093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.052075    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" event={"ID":"6cf1b65c-2559-4d34-9539-448e37119221","Type":"ContainerStarted","Data":"6140406f00c50b66e70a70fbffbd4aee78aaac1cae5181edac03702c19863200"}
Oct 30 19:38:28.058505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.057558    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" event={"ID":"137fbb43-62a3-415b-9cd9-15dad618d603","Type":"ContainerStarted","Data":"2513195b221f6d81a27ed894ec842e5e2b9c9bfd20f3927d095038eb2f6c209f"}
Oct 30 19:38:28.058505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.057597    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" event={"ID":"137fbb43-62a3-415b-9cd9-15dad618d603","Type":"ContainerStarted","Data":"4309beefbee1af7cbb0ae997082c908df7478329412a83c37416b05914690b06"}
Oct 30 19:38:28.060551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.060531    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8879-af7d4e" event={"ID":"bcbe8125-f207-441b-bb1f-2e776302c468","Type":"ContainerStarted","Data":"2ea39bb13c635e9b68f10baf799b1b13b7671fafd51daa3dd94d08297e93aa24"}
Oct 30 19:38:28.060681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.060671    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8879-af7d4e" event={"ID":"bcbe8125-f207-441b-bb1f-2e776302c468","Type":"ContainerStarted","Data":"a2cc0eabc0d71a4dba1dbc58a5eaa9a8fa416c0b373f42f9a7ee847c57ba1558"}
Oct 30 19:38:28.086945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.086890    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" podStartSLOduration=9.086876587 podStartE2EDuration="9.086876587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:28.078086337 +0000 UTC m=+1942.981099376" watchObservedRunningTime="2026-10-30 19:38:28.086876587 +0000 UTC m=+1942.989889623"
Oct 30 19:38:28.182843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.182797    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8879-af7d4e" podStartSLOduration=2.182781128 podStartE2EDuration="2.182781128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:28.132988778 +0000 UTC m=+1943.036001815" watchObservedRunningTime="2026-10-30 19:38:28.182781128 +0000 UTC m=+1943.085794166"
Oct 30 19:38:28.183313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.183283    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" podStartSLOduration=9.183274619 podStartE2EDuration="9.183274619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:28.156715473 +0000 UTC m=+1943.059728511" watchObservedRunningTime="2026-10-30 19:38:28.183274619 +0000 UTC m=+1943.086287655"
Oct 30 19:38:28.659740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.656224    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:28.874539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.874494    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:28.875078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.874646    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:38:28.875078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.874705    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:38:28.964896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.949353    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:28.965415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.965373    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:38:28.965636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:28.965604    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:38:29.288591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:29.288512    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8879-af7d4e"]
Oct 30 19:38:29.288857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:29.288841    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8879-af7d4e"]
Oct 30 19:38:29.694403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:29.694232    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:29.866136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:29.866097    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:38:29.866323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:29.866156    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:38:29.922713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:29.921908    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:38:29.922713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:29.921968    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:38:30.077571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:30.077493    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8879-af7d4e" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" containerName="test-curl-8879-af7d4e" containerID="cri-o://2ea39bb13c635e9b68f10baf799b1b13b7671fafd51daa3dd94d08297e93aa24" gracePeriod=2
Oct 30 19:38:30.080243 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:38:30.080212    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-af7d4e\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:30.816713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:30.816631    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:30.868435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:30.867763    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:38:30.868435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:30.867844    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:38:30.913798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:30.900192    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:38:30.913798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:30.900252    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:38:31.007937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:31.007901    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:31.872712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:31.867038    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:38:31.872712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:31.867108    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:38:31.902776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:31.893276    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:31.902776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:31.900379    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:38:31.902776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:31.900432    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:38:32.269894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:32.269861    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:32.640957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:32.640723    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:32.926761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:32.909275    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:38:32.926761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:32.909334    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:38:32.927973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:32.927227    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:38:32.927973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:32.927282    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:38:33.215715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.194614    4925 generic.go:356] "Generic (PLEG): container finished" podID="bcbe8125-f207-441b-bb1f-2e776302c468" containerID="2ea39bb13c635e9b68f10baf799b1b13b7671fafd51daa3dd94d08297e93aa24" exitCode=137
Oct 30 19:38:33.215715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.194771    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2cc0eabc0d71a4dba1dbc58a5eaa9a8fa416c0b373f42f9a7ee847c57ba1558"
Oct 30 19:38:33.343711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.340897    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:33.391735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.390933    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:33.547768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.547301    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbe8125-f207-441b-bb1f-2e776302c468-kube-api-access-tcd2b\" (UniqueName: \"kubernetes.io/projected/bcbe8125-f207-441b-bb1f-2e776302c468-kube-api-access-tcd2b\") pod \"bcbe8125-f207-441b-bb1f-2e776302c468\" (UID: \"bcbe8125-f207-441b-bb1f-2e776302c468\") "
Oct 30 19:38:33.561070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.561024    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbe8125-f207-441b-bb1f-2e776302c468-kube-api-access-tcd2b" pod "bcbe8125-f207-441b-bb1f-2e776302c468" (UID: "bcbe8125-f207-441b-bb1f-2e776302c468"). InnerVolumeSpecName "kube-api-access-tcd2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:33.652448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.652411    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcd2b\" (UniqueName: \"kubernetes.io/projected/bcbe8125-f207-441b-bb1f-2e776302c468-kube-api-access-tcd2b\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:33.873107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.872493    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:38:33.873107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.872550    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:38:33.899019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.898987    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:38:33.899280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:33.899252    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:38:34.050711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:34.050501    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:34.258001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:34.257968    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:34.275409 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:38:34.275369    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-af7d4e\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:34.800713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:34.798199    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:35.035615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:35.035552    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:35.037383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:35.037357    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:35.038298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:35.038278    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:38:35.039233 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:38:35.039203    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-af7d4e\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:35.067266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:35.067189    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:35.067536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:35.067514    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:35.068512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:35.068491    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:38:35.104918 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:38:35.104882    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-af7d4e\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:35.276740 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:38:35.276644    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-af7d4e\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" pod="opendatahub/test-curl-8879-af7d4e"
Oct 30 19:38:35.279912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:35.279883    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" path="/var/lib/kubelet/pods/bcbe8125-f207-441b-bb1f-2e776302c468/volumes"
Oct 30 19:38:35.843876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:35.843840    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:36.586506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:36.586466    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:37.033102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:37.033068    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"]
Oct 30 19:38:37.037511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:37.037485    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8c2e66-m4f4k"]
Oct 30 19:38:37.268001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:37.267962    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3146b621-6068-4071-9154-31fe79722b07" path="/var/lib/kubelet/pods/3146b621-6068-4071-9154-31fe79722b07/volumes"
Oct 30 19:38:37.364016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:37.363926    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:38.172548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:38.172512    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:38.793047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:38.793014    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:39.464966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:39.464933    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:40.355357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:40.355277    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:40.990751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:40.986705    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:41.717504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:41.717462    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:42.478015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:42.477966    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:43.178714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:43.172572    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:43.796486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:43.796449    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:44.675177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:44.675145    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:45.307195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:45.307135    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:45.895252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:45.895174    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:46.860930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:46.860891    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:47.483814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:47.483779    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:47.647751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:47.647711    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:48.494900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:48.494822    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:49.299916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:49.295308    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:50.175459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:50.175377    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:50.867792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:50.860587    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:51.585530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:51.585493    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:52.025720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:52.023258    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:52.161844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:52.161813    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"]
Oct 30 19:38:52.181436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:52.181391    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-995a9ec3-tlfvn"]
Oct 30 19:38:52.234082 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:38:52.230886    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:38:52.518596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:52.518568    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:52.693245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:52.693209    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:53.265426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:53.265393    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028c7f0d-a473-4753-8096-65af4f9264b5" path="/var/lib/kubelet/pods/028c7f0d-a473-4753-8096-65af4f9264b5/volumes"
Oct 30 19:38:53.451806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:53.445545    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:54.091450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:54.088064    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:54.702717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:54.691319    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:55.307973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:55.307940    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:55.485014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:55.484985    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:55.661736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:55.660090    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:56.437713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:56.435910    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:56.824022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:56.823934    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"]
Oct 30 19:38:56.927933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:56.927893    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:57.411489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:57.411451    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:57.829025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:57.828991    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:58.283396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:58.283269    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:58.969736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:58.967116    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:59.595342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:38:59.591582    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:01.470525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:01.470489    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:01.669882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:01.669846    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:02.054648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:02.054612    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:02.764786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:02.764749    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:03.555171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:03.553843    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:04.153371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:04.153333    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:05.029276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:05.029236    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:05.469465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:05.469342    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:06.225603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:06.225565    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:07.440727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:07.440670    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:07.910484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:07.910450    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:08.694364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:08.693739    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:09.559172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:09.559142    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:10.202713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:10.195972    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:10.639381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:10.639282    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:11.207913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:11.207871    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:11.862091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:11.862029    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" podUID="aeff9d2a-d624-4ab3-a316-13d796122ee2" containerName="main" containerID="cri-o://9aebc5e13a5de9e2cb3a1d49a39298366603354d5ea8b7f109d0197877543178" gracePeriod=45
Oct 30 19:39:11.906649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:11.906608    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" podUID="aeff9d2a-d624-4ab3-a316-13d796122ee2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.187:8000/ready\": read tcp 10.132.0.2:42030->10.132.0.187:8000: read: connection reset by peer"
Oct 30 19:39:11.921317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:11.921282    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:12.050441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.050410    4925 generic.go:356] "Generic (PLEG): container finished" podID="aeff9d2a-d624-4ab3-a316-13d796122ee2" containerID="9aebc5e13a5de9e2cb3a1d49a39298366603354d5ea8b7f109d0197877543178" exitCode=0
Oct 30 19:39:12.050641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.050498    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" event={"ID":"aeff9d2a-d624-4ab3-a316-13d796122ee2","Type":"ContainerDied","Data":"9aebc5e13a5de9e2cb3a1d49a39298366603354d5ea8b7f109d0197877543178"}
Oct 30 19:39:12.308983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.308957    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:39:12.421182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.421146    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeff9d2a-d624-4ab3-a316-13d796122ee2-kube-api-access-zhxrx\" (UniqueName: \"kubernetes.io/projected/aeff9d2a-d624-4ab3-a316-13d796122ee2-kube-api-access-zhxrx\") pod \"aeff9d2a-d624-4ab3-a316-13d796122ee2\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") "
Oct 30 19:39:12.421376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.421203    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeff9d2a-d624-4ab3-a316-13d796122ee2-tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff9d2a-d624-4ab3-a316-13d796122ee2-tls-certs\") pod \"aeff9d2a-d624-4ab3-a316-13d796122ee2\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") "
Oct 30 19:39:12.421376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.421244    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-dshm\") pod \"aeff9d2a-d624-4ab3-a316-13d796122ee2\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") "
Oct 30 19:39:12.421376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.421305    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-home\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-home\") pod \"aeff9d2a-d624-4ab3-a316-13d796122ee2\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") "
Oct 30 19:39:12.421660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.421636    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-tmp-dir\") pod \"aeff9d2a-d624-4ab3-a316-13d796122ee2\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") "
Oct 30 19:39:12.421824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.421655    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-home" pod "aeff9d2a-d624-4ab3-a316-13d796122ee2" (UID: "aeff9d2a-d624-4ab3-a316-13d796122ee2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:12.421824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.421712    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-model-cache\") pod \"aeff9d2a-d624-4ab3-a316-13d796122ee2\" (UID: \"aeff9d2a-d624-4ab3-a316-13d796122ee2\") "
Oct 30 19:39:12.422127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.421905    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-model-cache" pod "aeff9d2a-d624-4ab3-a316-13d796122ee2" (UID: "aeff9d2a-d624-4ab3-a316-13d796122ee2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:12.422127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.421913    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-tmp-dir" pod "aeff9d2a-d624-4ab3-a316-13d796122ee2" (UID: "aeff9d2a-d624-4ab3-a316-13d796122ee2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:12.422941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.422913    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.423062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.422942    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.423062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.422957    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.423882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.423860    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeff9d2a-d624-4ab3-a316-13d796122ee2-tls-certs" pod "aeff9d2a-d624-4ab3-a316-13d796122ee2" (UID: "aeff9d2a-d624-4ab3-a316-13d796122ee2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:12.423966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.423904    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeff9d2a-d624-4ab3-a316-13d796122ee2-kube-api-access-zhxrx" pod "aeff9d2a-d624-4ab3-a316-13d796122ee2" (UID: "aeff9d2a-d624-4ab3-a316-13d796122ee2"). InnerVolumeSpecName "kube-api-access-zhxrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:12.424025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.423968    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-dshm" pod "aeff9d2a-d624-4ab3-a316-13d796122ee2" (UID: "aeff9d2a-d624-4ab3-a316-13d796122ee2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:12.525193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.525125    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhxrx\" (UniqueName: \"kubernetes.io/projected/aeff9d2a-d624-4ab3-a316-13d796122ee2-kube-api-access-zhxrx\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.525193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.525161    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff9d2a-d624-4ab3-a316-13d796122ee2-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.525193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.525175    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff9d2a-d624-4ab3-a316-13d796122ee2-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.690668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:12.690631    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:13.060529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:13.060503    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"
Oct 30 19:39:13.060765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:13.060547    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj" event={"ID":"aeff9d2a-d624-4ab3-a316-13d796122ee2","Type":"ContainerDied","Data":"5a27f7b7d7382331cf03872b5cf8d3f90ec926a8163ecea9eed27f726f3933a4"}
Oct 30 19:39:13.060765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:13.060597    4925 scope.go:122] "RemoveContainer" containerID="9aebc5e13a5de9e2cb3a1d49a39298366603354d5ea8b7f109d0197877543178"
Oct 30 19:39:13.089533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:13.089507    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"]
Oct 30 19:39:13.093868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:13.093832    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/auth-test-model-0a6e66-kserve-798896b49c-j2dqj"]
Oct 30 19:39:13.273921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:13.272560    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeff9d2a-d624-4ab3-a316-13d796122ee2" path="/var/lib/kubelet/pods/aeff9d2a-d624-4ab3-a316-13d796122ee2/volumes"
Oct 30 19:39:13.476714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:13.475609    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:14.448710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:14.448064    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:15.217622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:15.217034    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:15.912242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:15.912205    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:17.024502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:17.024023    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:17.779598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:17.779560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:18.064806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:18.064730    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"]
Oct 30 19:39:18.177670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:18.177637    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:19.055726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:19.054103    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:19.863802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:19.863765    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:20.690045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:20.689790    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:21.526133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:21.526095    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:22.129292 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:39:22.129252    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:39:22.320302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:22.320267    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:23.038248 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:23.038206    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"]
Oct 30 19:39:23.059713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:23.054169    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40dab794-vvf58"]
Oct 30 19:39:23.088544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:23.087754    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:23.265432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:23.265390    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd52578e-8671-4cda-b6f1-c08931f772a6" path="/var/lib/kubelet/pods/bd52578e-8671-4cda-b6f1-c08931f772a6/volumes"
Oct 30 19:39:23.793710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:23.786290    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:24.514783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:24.513260    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:25.459666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:25.459445    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:25.815969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:25.814609    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"]
Oct 30 19:39:25.947743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:25.947316    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:26.387204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:26.387168    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:27.137730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:27.133264    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:28.054206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:28.054169    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:28.380756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:28.380667    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"]
Oct 30 19:39:28.418293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:28.418258    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"]
Oct 30 19:39:28.521227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:28.521198    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:28.690297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:28.690260    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:29.123815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:29.123602    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:29.883712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:29.879209    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:30.682894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:30.682860    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:31.377492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:31.377443    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:32.002866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:32.002831    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:32.975709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:32.975659    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:33.087200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.087155    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main" containerID="cri-o://5636b13173e78216dc381b680fb363bb1bc72345c1b4465bdb10d8c267998dce" gracePeriod=45
Oct 30 19:39:33.247261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.247166    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": dial tcp 10.132.0.182:8000: connect: connection refused"
Oct 30 19:39:33.264134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.264084    4925 generic.go:356] "Generic (PLEG): container finished" podID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerID="5636b13173e78216dc381b680fb363bb1bc72345c1b4465bdb10d8c267998dce" exitCode=0
Oct 30 19:39:33.273635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.273602    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" event={"ID":"a62f41cc-7f6d-4be3-b26d-24c474350cb5","Type":"ContainerDied","Data":"5636b13173e78216dc381b680fb363bb1bc72345c1b4465bdb10d8c267998dce"}
Oct 30 19:39:33.381973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.381930    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb" podUID="96cd40c7-a0aa-48cc-8e19-e5d626cd6d15" containerName="payload-processing" containerID="cri-o://cd57674fd299a454a42b4eb8edaf0c8857030b03596aa044dac691f1e4905c0c" gracePeriod=25
Oct 30 19:39:33.419224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.419188    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz" podUID="2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" containerName="payload-pre-processing" containerID="cri-o://e37adc3b266b78fb899fc98e600928a8120e659bdb3e35a94fb3e63813529c21" gracePeriod=25
Oct 30 19:39:33.462451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.462398    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"]
Oct 30 19:39:33.463346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.462949    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm" podUID="7c324c78-be8e-430e-adc9-3827a16a8dba" containerName="maas-api" containerID="cri-o://fabb01223cf37092c16f87c1dcb8dfc582bb8bddf688ba5c1591b814dd42f9aa" gracePeriod=30
Oct 30 19:39:33.483196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.483165    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:39:33.566090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.566020    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:33.598483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.598440    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62f41cc-7f6d-4be3-b26d-24c474350cb5-kube-api-access-xvxsw\" (UniqueName: \"kubernetes.io/projected/a62f41cc-7f6d-4be3-b26d-24c474350cb5-kube-api-access-xvxsw\") pod \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") "
Oct 30 19:39:33.600747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.598930    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-dshm\") pod \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") "
Oct 30 19:39:33.600747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.599101    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-model-cache\") pod \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") "
Oct 30 19:39:33.600747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.599223    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tmp-dir\") pod \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") "
Oct 30 19:39:33.600747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.599252    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-home\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-home\") pod \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") "
Oct 30 19:39:33.600747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.599329    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tls-certs\") pod \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\" (UID: \"a62f41cc-7f6d-4be3-b26d-24c474350cb5\") "
Oct 30 19:39:33.600747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.599888    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tmp-dir" pod "a62f41cc-7f6d-4be3-b26d-24c474350cb5" (UID: "a62f41cc-7f6d-4be3-b26d-24c474350cb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.600747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.600500    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:33.600747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.600603    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-home" pod "a62f41cc-7f6d-4be3-b26d-24c474350cb5" (UID: "a62f41cc-7f6d-4be3-b26d-24c474350cb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.600747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.600716    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-model-cache" pod "a62f41cc-7f6d-4be3-b26d-24c474350cb5" (UID: "a62f41cc-7f6d-4be3-b26d-24c474350cb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.629709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.627852    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-dshm" pod "a62f41cc-7f6d-4be3-b26d-24c474350cb5" (UID: "a62f41cc-7f6d-4be3-b26d-24c474350cb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:33.629709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.628321    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tls-certs" pod "a62f41cc-7f6d-4be3-b26d-24c474350cb5" (UID: "a62f41cc-7f6d-4be3-b26d-24c474350cb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:33.633153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.633116    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62f41cc-7f6d-4be3-b26d-24c474350cb5-kube-api-access-xvxsw" pod "a62f41cc-7f6d-4be3-b26d-24c474350cb5" (UID: "a62f41cc-7f6d-4be3-b26d-24c474350cb5"). InnerVolumeSpecName "kube-api-access-xvxsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:33.779898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.702804    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:33.779898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.702840    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:33.779898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.702853    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62f41cc-7f6d-4be3-b26d-24c474350cb5-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:33.779898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.702870    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvxsw\" (UniqueName: \"kubernetes.io/projected/a62f41cc-7f6d-4be3-b26d-24c474350cb5-kube-api-access-xvxsw\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:33.779898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.702885    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62f41cc-7f6d-4be3-b26d-24c474350cb5-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:33.929816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:33.897816    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:34.188224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.119793    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:34.221053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.220997    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"]
Oct 30 19:39:34.259801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.259260    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz" podUID="2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:39:34.277980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.272172    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227c58e7-l9gbm"]
Oct 30 19:39:34.294096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.294063    4925 generic.go:356] "Generic (PLEG): container finished" podID="96cd40c7-a0aa-48cc-8e19-e5d626cd6d15" containerID="cd57674fd299a454a42b4eb8edaf0c8857030b03596aa044dac691f1e4905c0c" exitCode=0
Oct 30 19:39:34.294411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.294385    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb" event={"ID":"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15","Type":"ContainerDied","Data":"cd57674fd299a454a42b4eb8edaf0c8857030b03596aa044dac691f1e4905c0c"}
Oct 30 19:39:34.297235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.297207    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4" event={"ID":"a62f41cc-7f6d-4be3-b26d-24c474350cb5","Type":"ContainerDied","Data":"8a1746fd1ad8bc48ab5a8a5357e7b08d676899a184101ba92a4c0fa9522d59d9"}
Oct 30 19:39:34.297349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.297259    4925 scope.go:122] "RemoveContainer" containerID="5636b13173e78216dc381b680fb363bb1bc72345c1b4465bdb10d8c267998dce"
Oct 30 19:39:34.297982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.297961    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"
Oct 30 19:39:34.305061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.305014    4925 generic.go:356] "Generic (PLEG): container finished" podID="7c324c78-be8e-430e-adc9-3827a16a8dba" containerID="fabb01223cf37092c16f87c1dcb8dfc582bb8bddf688ba5c1591b814dd42f9aa" exitCode=0
Oct 30 19:39:34.305061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.305056    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm" event={"ID":"7c324c78-be8e-430e-adc9-3827a16a8dba","Type":"ContainerDied","Data":"fabb01223cf37092c16f87c1dcb8dfc582bb8bddf688ba5c1591b814dd42f9aa"}
Oct 30 19:39:34.345925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.345896    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:39:34.356972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.356944    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"]
Oct 30 19:39:34.362867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.362844    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1fa500/test-model-1fa500-kserve-8f96478d5-q7mz4"]
Oct 30 19:39:34.390118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.390090    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"]
Oct 30 19:39:34.391072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391049    4925 cpu_manager.go:414] "containerMap: removing container" podUID="aeff9d2a-d624-4ab3-a316-13d796122ee2" containerName="main"
Oct 30 19:39:34.391072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391072    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeff9d2a-d624-4ab3-a316-13d796122ee2" containerName="main"
Oct 30 19:39:34.391259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391105    4925 cpu_manager.go:414] "containerMap: removing container" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main"
Oct 30 19:39:34.391259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391113    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main"
Oct 30 19:39:34.391259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391186    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" containerName="test-curl-8879-af7d4e"
Oct 30 19:39:34.391259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391194    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" containerName="test-curl-8879-af7d4e"
Oct 30 19:39:34.391259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391213    4925 cpu_manager.go:414] "containerMap: removing container" podUID="96cd40c7-a0aa-48cc-8e19-e5d626cd6d15" containerName="payload-processing"
Oct 30 19:39:34.391259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391222    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cd40c7-a0aa-48cc-8e19-e5d626cd6d15" containerName="payload-processing"
Oct 30 19:39:34.391568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391399    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbe8125-f207-441b-bb1f-2e776302c468" containerName="test-curl-8879-af7d4e"
Oct 30 19:39:34.391568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391416    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cd40c7-a0aa-48cc-8e19-e5d626cd6d15" containerName="payload-processing"
Oct 30 19:39:34.391568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391430    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeff9d2a-d624-4ab3-a316-13d796122ee2" containerName="main"
Oct 30 19:39:34.391568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.391443    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" containerName="main"
Oct 30 19:39:34.400741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.398279    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:34.401725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.401599    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-178bcc\""
Oct 30 19:39:34.402482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.402460    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-178bcc-dockercfg-6rr7f\""
Oct 30 19:39:34.405616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.405594    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"]
Oct 30 19:39:34.424394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.424342    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-kube-api-access-s6kk2\" (UniqueName: \"kubernetes.io/projected/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-kube-api-access-s6kk2\") pod \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\" (UID: \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\") "
Oct 30 19:39:34.424883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.424859    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-plugins-config-volume\") pod \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\" (UID: \"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15\") "
Oct 30 19:39:34.427139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.425962    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-plugins-config-volume" pod "96cd40c7-a0aa-48cc-8e19-e5d626cd6d15" (UID: "96cd40c7-a0aa-48cc-8e19-e5d626cd6d15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:34.427476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.427458    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:34.428443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.428405    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-kube-api-access-s6kk2" pod "96cd40c7-a0aa-48cc-8e19-e5d626cd6d15" (UID: "96cd40c7-a0aa-48cc-8e19-e5d626cd6d15"). InnerVolumeSpecName "kube-api-access-s6kk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:34.445369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.445305    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:39:34.447194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.447167    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:34.486932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.486903    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"]
Oct 30 19:39:34.487671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.487656    4925 cpu_manager.go:414] "containerMap: removing container" podUID="7c324c78-be8e-430e-adc9-3827a16a8dba" containerName="maas-api"
Oct 30 19:39:34.487735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.487674    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c324c78-be8e-430e-adc9-3827a16a8dba" containerName="maas-api"
Oct 30 19:39:34.487923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.487908    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c324c78-be8e-430e-adc9-3827a16a8dba" containerName="maas-api"
Oct 30 19:39:34.491794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.491770    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.507027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.507002    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"]
Oct 30 19:39:34.529043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.529012    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-metrics-certs\") pod \"7c324c78-be8e-430e-adc9-3827a16a8dba\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") "
Oct 30 19:39:34.529222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.529078    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-maas-api-tls\") pod \"7c324c78-be8e-430e-adc9-3827a16a8dba\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") "
Oct 30 19:39:34.529222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.529197    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c324c78-be8e-430e-adc9-3827a16a8dba-kube-api-access-dfmvc\" (UniqueName: \"kubernetes.io/projected/7c324c78-be8e-430e-adc9-3827a16a8dba-kube-api-access-dfmvc\") pod \"7c324c78-be8e-430e-adc9-3827a16a8dba\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") "
Oct 30 19:39:34.529424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.529296    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c324c78-be8e-430e-adc9-3827a16a8dba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c324c78-be8e-430e-adc9-3827a16a8dba-openshift-service-ca\") pod \"7c324c78-be8e-430e-adc9-3827a16a8dba\" (UID: \"7c324c78-be8e-430e-adc9-3827a16a8dba\") "
Oct 30 19:39:34.529730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.529654    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c324c78-be8e-430e-adc9-3827a16a8dba-openshift-service-ca" pod "7c324c78-be8e-430e-adc9-3827a16a8dba" (UID: "7c324c78-be8e-430e-adc9-3827a16a8dba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:34.530025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.529854    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7fn\" (UniqueName: \"kubernetes.io/projected/22843a64-604b-41bf-b6ea-520f5246159b-kube-api-access-pc7fn\") pod \"payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr\" (UID: \"22843a64-604b-41bf-b6ea-520f5246159b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:34.530224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.530198    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22843a64-604b-41bf-b6ea-520f5246159b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr\" (UID: \"22843a64-604b-41bf-b6ea-520f5246159b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:34.530887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.530861    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6kk2\" (UniqueName: \"kubernetes.io/projected/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15-kube-api-access-s6kk2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:34.531005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.530892    4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c324c78-be8e-430e-adc9-3827a16a8dba-openshift-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:34.531786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.531759    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-metrics-certs" pod "7c324c78-be8e-430e-adc9-3827a16a8dba" (UID: "7c324c78-be8e-430e-adc9-3827a16a8dba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:34.531877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.531829    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c324c78-be8e-430e-adc9-3827a16a8dba-kube-api-access-dfmvc" pod "7c324c78-be8e-430e-adc9-3827a16a8dba" (UID: "7c324c78-be8e-430e-adc9-3827a16a8dba"). InnerVolumeSpecName "kube-api-access-dfmvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:34.531877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.531857    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-maas-api-tls" pod "7c324c78-be8e-430e-adc9-3827a16a8dba" (UID: "7c324c78-be8e-430e-adc9-3827a16a8dba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:34.632641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.632599    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22843a64-604b-41bf-b6ea-520f5246159b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr\" (UID: \"22843a64-604b-41bf-b6ea-520f5246159b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:34.632949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.632661    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-certs\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.632949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.632804    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j647l\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-kube-api-access-j647l\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.633083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.632946    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-token\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.633083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633027    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-credential-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.633083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633054    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-podinfo\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.633195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633153    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.633244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633199    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-envoy\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.633244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633231    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7fn\" (UniqueName: \"kubernetes.io/projected/22843a64-604b-41bf-b6ea-520f5246159b-kube-api-access-pc7fn\") pod \"payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr\" (UID: \"22843a64-604b-41bf-b6ea-520f5246159b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:34.633424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633397    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istiod-ca-cert\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.633510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633429    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22843a64-604b-41bf-b6ea-520f5246159b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr\" (UID: \"22843a64-604b-41bf-b6ea-520f5246159b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:34.633510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633458    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-data\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.633637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633619    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfmvc\" (UniqueName: \"kubernetes.io/projected/7c324c78-be8e-430e-adc9-3827a16a8dba-kube-api-access-dfmvc\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:34.633677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633648    4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-metrics-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:34.633677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.633664    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c324c78-be8e-430e-adc9-3827a16a8dba-maas-api-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:34.649705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.649664    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7fn\" (UniqueName: \"kubernetes.io/projected/22843a64-604b-41bf-b6ea-520f5246159b-kube-api-access-pc7fn\") pod \"payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr\" (UID: \"22843a64-604b-41bf-b6ea-520f5246159b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:34.735725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.735610    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.735725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.735702    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-envoy\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.735941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.735753    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istiod-ca-cert\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.735941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.735781    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-data\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736019    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-certs\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736069    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736206    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-data\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736300    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j647l\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-kube-api-access-j647l\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736329    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-certs\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736388    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-token\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736437    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-credential-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736463    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-podinfo\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736509    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istiod-ca-cert\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.736952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.736922    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-credential-socket\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.738278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.738257    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-envoy\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.738967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.738950    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-podinfo\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.748258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.748235    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-token\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.748382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.748322    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j647l\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-kube-api-access-j647l\") pod \"e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") " pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:34.758619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.758597    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:34.812207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:34.812179    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:35.056216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.056181    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"]
Oct 30 19:39:35.110524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.110458    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:35.154000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.153059    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"]
Oct 30 19:39:35.155866 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:39:35.155827    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9117e1cf_fa3d_43a2_969f_7d0baa2a2e6c.slice/crio-96cb0d29c436608ed294382eb75a235b9de392dc841395270e65637fce597079 WatchSource:0}: Error finding container 96cb0d29c436608ed294382eb75a235b9de392dc841395270e65637fce597079: Status 404 returned error can't find the container with id 96cb0d29c436608ed294382eb75a235b9de392dc841395270e65637fce597079
Oct 30 19:39:35.158997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.158954    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:35.159098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.159041    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:35.159098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.159082    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:35.185957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.185926    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb" podUID="96cd40c7-a0aa-48cc-8e19-e5d626cd6d15" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.159:50052\" within 1s: context deadline exceeded"
Oct 30 19:39:35.186088 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:39:35.186032    4925 logging.go:55] [core] [Channel #1942 SubChannel #1943]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.159:50052", ServerName: "10.132.0.159:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.159:50052: operation was canceled"
Oct 30 19:39:35.268091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.268060    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62f41cc-7f6d-4be3-b26d-24c474350cb5" path="/var/lib/kubelet/pods/a62f41cc-7f6d-4be3-b26d-24c474350cb5/volumes"
Oct 30 19:39:35.268799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.268778    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4959ec-c75d-42e9-95ca-433649f375bb" path="/var/lib/kubelet/pods/bf4959ec-c75d-42e9-95ca-433649f375bb/volumes"
Oct 30 19:39:35.325748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.325615    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"
Oct 30 19:39:35.325748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.325625    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb" event={"ID":"96cd40c7-a0aa-48cc-8e19-e5d626cd6d15","Type":"ContainerDied","Data":"eaeae552b75fad713558bfcb5954c250f515cf4201fec6e8873c33267072e738"}
Oct 30 19:39:35.325969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.325773    4925 scope.go:122] "RemoveContainer" containerID="cd57674fd299a454a42b4eb8edaf0c8857030b03596aa044dac691f1e4905c0c"
Oct 30 19:39:35.328131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.328106    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" event={"ID":"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c","Type":"ContainerStarted","Data":"634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db"}
Oct 30 19:39:35.328254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.328155    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" event={"ID":"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c","Type":"ContainerStarted","Data":"96cb0d29c436608ed294382eb75a235b9de392dc841395270e65637fce597079"}
Oct 30 19:39:35.330249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.330219    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr" event={"ID":"22843a64-604b-41bf-b6ea-520f5246159b","Type":"ContainerStarted","Data":"6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d"}
Oct 30 19:39:35.330418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.330271    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr" event={"ID":"22843a64-604b-41bf-b6ea-520f5246159b","Type":"ContainerStarted","Data":"b38f54f11ba663c25dbb13a759c2ce2fd6de0c257b679c812356ce7b81e1e2ab"}
Oct 30 19:39:35.330418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.330301    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:35.333172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.333152    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"
Oct 30 19:39:35.333326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.333178    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm" event={"ID":"7c324c78-be8e-430e-adc9-3827a16a8dba","Type":"ContainerDied","Data":"3619aba341547aec1bb8b5a5449332ab86e0bc1a4d62453b8a97b5a496c09aad"}
Oct 30 19:39:35.342515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.342496    4925 scope.go:122] "RemoveContainer" containerID="fabb01223cf37092c16f87c1dcb8dfc582bb8bddf688ba5c1591b814dd42f9aa"
Oct 30 19:39:35.359399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.359345    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podStartSLOduration=43.359328401 podStartE2EDuration="43.359328401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:35.353279767 +0000 UTC m=+2010.256292803" watchObservedRunningTime="2026-10-30 19:39:35.359328401 +0000 UTC m=+2010.262341438"
Oct 30 19:39:35.375990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.375943    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr" podStartSLOduration=63.375926813 podStartE2EDuration="1m3.375926813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:35.374312031 +0000 UTC m=+2010.277325068" watchObservedRunningTime="2026-10-30 19:39:35.375926813 +0000 UTC m=+2010.278939853"
Oct 30 19:39:35.396136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.396108    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"]
Oct 30 19:39:35.397377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.397354    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2dbd9-7cb7fcf656-rxnzm"]
Oct 30 19:39:35.411640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.411614    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"]
Oct 30 19:39:35.421031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.421010    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2dbd9-7b9996c75d-gnthb"]
Oct 30 19:39:35.813185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.813146    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:35.832976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:35.832934    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:36.039745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:36.022411    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"]
Oct 30 19:39:36.043503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:36.043473    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:36.273714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:36.269934    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:36.750045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:36.750014    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:36.819990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:36.819799    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:36.819990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:36.819864    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:37.260717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:37.245363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:37.426717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:37.425099    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c324c78-be8e-430e-adc9-3827a16a8dba" path="/var/lib/kubelet/pods/7c324c78-be8e-430e-adc9-3827a16a8dba/volumes"
Oct 30 19:39:37.443712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:37.439796    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96cd40c7-a0aa-48cc-8e19-e5d626cd6d15" path="/var/lib/kubelet/pods/96cd40c7-a0aa-48cc-8e19-e5d626cd6d15/volumes"
Oct 30 19:39:37.653026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:37.640299    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:37.832655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:37.828294    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:37.832655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:37.828347    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:38.202786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:38.198132    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:38.813714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:38.812995    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:38.813714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:38.813067    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:38.941794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:38.941759    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:39.788200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:39.788009    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:39.819533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:39.819488    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:39.820056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:39.819567    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:40.519404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:40.519367    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:40.815124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:40.815032    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:40.815399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:40.815365    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:40.861210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:40.857739    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" containerID="cri-o://ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6" gracePeriod=45
Oct 30 19:39:40.960524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:40.959401    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": EOF"
Oct 30 19:39:41.242114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.242081    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:39:41.284157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.284121    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:41.342079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.341994    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"]
Oct 30 19:39:41.343152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.343129    4925 cpu_manager.go:414] "containerMap: removing container" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main"
Oct 30 19:39:41.343318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.343301    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main"
Oct 30 19:39:41.343781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.343760    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerName="main"
Oct 30 19:39:41.365429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.365399    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.373563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.373520    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0a6e66\"/\"default-dockercfg-xfs4c\""
Oct 30 19:39:41.373784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.373648    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0a6e66\"/\"sub-test-model-0a6e66-kserve-self-signed-certs\""
Oct 30 19:39:41.373996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.373950    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"]
Oct 30 19:39:41.382301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.382177    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-home\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-home\") pod \"399ffc7e-58f7-4662-9745-c2feddef06cb\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") "
Oct 30 19:39:41.382499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.382324    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-dshm\") pod \"399ffc7e-58f7-4662-9745-c2feddef06cb\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") "
Oct 30 19:39:41.382499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.382460    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-model-cache\") pod \"399ffc7e-58f7-4662-9745-c2feddef06cb\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") "
Oct 30 19:39:41.382749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.382651    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399ffc7e-58f7-4662-9745-c2feddef06cb-kube-api-access-gnzmf\" (UniqueName: \"kubernetes.io/projected/399ffc7e-58f7-4662-9745-c2feddef06cb-kube-api-access-gnzmf\") pod \"399ffc7e-58f7-4662-9745-c2feddef06cb\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") "
Oct 30 19:39:41.382749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.382726    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-tmp-dir\") pod \"399ffc7e-58f7-4662-9745-c2feddef06cb\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") "
Oct 30 19:39:41.382955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.382756    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/399ffc7e-58f7-4662-9745-c2feddef06cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/399ffc7e-58f7-4662-9745-c2feddef06cb-tls-certs\") pod \"399ffc7e-58f7-4662-9745-c2feddef06cb\" (UID: \"399ffc7e-58f7-4662-9745-c2feddef06cb\") "
Oct 30 19:39:41.384457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.384367    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-model-cache" pod "399ffc7e-58f7-4662-9745-c2feddef06cb" (UID: "399ffc7e-58f7-4662-9745-c2feddef06cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:41.385173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.384903    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-tmp-dir" pod "399ffc7e-58f7-4662-9745-c2feddef06cb" (UID: "399ffc7e-58f7-4662-9745-c2feddef06cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:41.385173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.385161    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-home" pod "399ffc7e-58f7-4662-9745-c2feddef06cb" (UID: "399ffc7e-58f7-4662-9745-c2feddef06cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:41.391513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.390543    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399ffc7e-58f7-4662-9745-c2feddef06cb-kube-api-access-gnzmf" pod "399ffc7e-58f7-4662-9745-c2feddef06cb" (UID: "399ffc7e-58f7-4662-9745-c2feddef06cb"). InnerVolumeSpecName "kube-api-access-gnzmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:41.415713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.407121    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0a6e66\"/\"kube-root-ca.crt\""
Oct 30 19:39:41.415713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.407295    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0a6e66\"/\"openshift-service-ca.crt\""
Oct 30 19:39:41.415713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.408110    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399ffc7e-58f7-4662-9745-c2feddef06cb-tls-certs" pod "399ffc7e-58f7-4662-9745-c2feddef06cb" (UID: "399ffc7e-58f7-4662-9745-c2feddef06cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:41.416043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.415809    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-dshm" pod "399ffc7e-58f7-4662-9745-c2feddef06cb" (UID: "399ffc7e-58f7-4662-9745-c2feddef06cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:41.475145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.475117    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:41.485270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485233    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hmrr\" (UniqueName: \"kubernetes.io/projected/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-kube-api-access-8hmrr\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.485473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485452    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tmp-dir\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.485596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485579    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-home\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.485670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485641    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-dshm\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.485670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485647    4925 generic.go:356] "Generic (PLEG): container finished" podID="399ffc7e-58f7-4662-9745-c2feddef06cb" containerID="ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6" exitCode=0
Oct 30 19:39:41.485842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485733    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tls-certs\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.485842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485742    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"
Oct 30 19:39:41.485842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485755    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" event={"ID":"399ffc7e-58f7-4662-9745-c2feddef06cb","Type":"ContainerDied","Data":"ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6"}
Oct 30 19:39:41.485842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485801    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv" event={"ID":"399ffc7e-58f7-4662-9745-c2feddef06cb","Type":"ContainerDied","Data":"0b657e6c63cafaf820716c4ca8c7e2dc9389b2c2de541d0c0d512440e7b1e425"}
Oct 30 19:39:41.485842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485824    4925 scope.go:122] "RemoveContainer" containerID="ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6"
Oct 30 19:39:41.485842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.485823    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-model-cache\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.486297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.486275    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnzmf\" (UniqueName: \"kubernetes.io/projected/399ffc7e-58f7-4662-9745-c2feddef06cb-kube-api-access-gnzmf\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.486374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.486302    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.486374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.486317    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/399ffc7e-58f7-4662-9745-c2feddef06cb-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.486374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.486331    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.486374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.486345    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.486374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.486359    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/399ffc7e-58f7-4662-9745-c2feddef06cb-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.541583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.541469    4925 scope.go:122] "RemoveContainer" containerID="ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6"
Oct 30 19:39:41.543943 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:39:41.543831    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6\": container with ID starting with ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6 not found: ID does not exist" containerID="ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6"
Oct 30 19:39:41.543943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.543872    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6"} err="failed to get container status \"ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6\": rpc error: code = NotFound desc = could not find container \"ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6\": container with ID starting with ae99a33d932912ab86c32ed4dbf82f69e8cc1c495a2da73630905af1823f53c6 not found: ID does not exist"
Oct 30 19:39:41.563516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.563489    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"]
Oct 30 19:39:41.568918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.568895    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-618d93/rate-test-model-618d93-kserve-7c77cd7bc7-kqqrv"]
Oct 30 19:39:41.587611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.587590    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-home\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.587725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.587635    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-dshm\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.587725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.587661    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tls-certs\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.587845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.587751    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-model-cache\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.587943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.587915    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmrr\" (UniqueName: \"kubernetes.io/projected/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-kube-api-access-8hmrr\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.588049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.588036    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tmp-dir\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.588119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.588063    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-home\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.588402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.588380    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tmp-dir\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.588483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.588430    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-model-cache\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.590248 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.590214    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-dshm\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.590909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.590891    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tls-certs\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.617764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.610392    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmrr\" (UniqueName: \"kubernetes.io/projected/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-kube-api-access-8hmrr\") pod \"sub-test-model-0a6e66-kserve-58d5697fd9-kvw69\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") " pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.698710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.698049    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:41.815709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.813160    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:41.815709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:41.813217    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:42.708711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.707808    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.763711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.761443    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:42.773712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.764804    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.773712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.771643    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.773712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.772027    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.831717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.814098    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:42.831717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.814164    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:42.833869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.832080    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.872842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.872784    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.873347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.873316    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:39:42.873836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.873810    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded"
Oct 30 19:39:42.877710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.876296    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:39:42.930715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.930200    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.955308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.954434    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.955578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.955551    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.985022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:42.974784    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:43.056616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.045992    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"]
Oct 30 19:39:43.058924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.058888    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy" containerID="cri-o://bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99" gracePeriod=30
Oct 30 19:39:43.224842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.224803    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:43.254350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.253086    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.281298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.281250    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Oct 30 19:39:43.281657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.281631    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.314940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.313837    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.387716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.362220    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.449720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.430770    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.497014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.496970    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.615760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.602874    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399ffc7e-58f7-4662-9745-c2feddef06cb" path="/var/lib/kubelet/pods/399ffc7e-58f7-4662-9745-c2feddef06cb/volumes"
Oct 30 19:39:43.687739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.686978    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.687739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.687254    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 19:39:43.710829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.710780    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.712050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.712027    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:39:43.788743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.772789    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.788743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.773197    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.788743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.773864    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:39:43.788743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.773934    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:39:43.788743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.775771    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.788743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.776360    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:39:43.814727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.799298    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"]
Oct 30 19:39:43.818725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.815338    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:43.818725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.815395    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:43.871766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.832214    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.872886 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.872814    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:39:43.873259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.867712    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:43.916715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.882079    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.916715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.882625    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:39:43.916715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.883166    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:43.916715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.883421    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.916715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.883637    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.932852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.923214    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:39:43.932852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.923272    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:39:43.932852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.923311    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:39:43.934671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.933511    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:39:43.934671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.934120    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:39:43.982782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:43.946296    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:44.016711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.012151    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.039930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.018812    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.048731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.045944    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:39:44.048731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.046393    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:39:44.057743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.056217    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:39:44.120763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.079086    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcdp\" (UniqueName: \"kubernetes.io/projected/c491d703-c607-45f6-958c-e3f1a942ed62-kube-api-access-glcdp\") pod \"maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:44.120763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.079533    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:44.120763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.079615    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:44.314933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.300574    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.314933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.300962    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.314933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.301512    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:39:44.314933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.301575    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:39:44.386039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.364420    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:44.400653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.400610    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.463864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.438829    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.463864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.439535    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:39:44.463864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.439602    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:39:44.463864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.443944    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/health\": context deadline exceeded"
Oct 30 19:39:44.602816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.534559    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:44.602816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.534680    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:44.602816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.534775    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcdp\" (UniqueName: \"kubernetes.io/projected/c491d703-c607-45f6-958c-e3f1a942ed62-kube-api-access-glcdp\") pod \"maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:44.638743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.604336    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.638743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.612818    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:39:44.649921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.647506    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:44.649921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.647932    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.671077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.669944    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz" podUID="2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:39:44.724862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.724396    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcdp\" (UniqueName: \"kubernetes.io/projected/c491d703-c607-45f6-958c-e3f1a942ed62-kube-api-access-glcdp\") pod \"maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:44.757770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.757730    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.841714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.838267    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.872867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.872763    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.894580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.893466    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.894897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.894867    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.918034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.917988    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.932772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.923197    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.932772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.923234    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded"
Oct 30 19:39:44.932772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.923264    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.939044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.937367    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:44.983503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:44.983470    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:45.053716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.006215    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:45.053716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.012889    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:45.053716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.012759    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.054820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.054777    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.083966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.083924    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:39:45.086989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.086939    4925 patch_prober.go:29] interesting pod/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:39:45.087538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.087510    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:39:45.421710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.412263    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.480840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.480789    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Oct 30 19:39:45.624730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.613000    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.818475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.625859    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.821144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.731918    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:39:45.835723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.826151    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:45.888136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.888092    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:45.915140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:45.915100    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.023807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.010193    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.023807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.015722    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Oct 30 19:39:46.023807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.015858    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.023807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.020792    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.023807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.022406    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.047466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.031254    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.271680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.266073    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:46.402430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.303810    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.402430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.389025    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.603260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.513576    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:46.603260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.513654    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:46.603260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.514473    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.603260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.514664    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.603260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.514861    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Oct 30 19:39:46.877837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.877732    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:46.886756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.886675    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:46.899775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.889468    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.903866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.902213    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.920552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.911768    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.937467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.933423    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.946368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.946326    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.947919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.947891    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.959710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.950638    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:39:46.966260 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:39:46.966216    4925 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.444s"
Oct 30 19:39:46.966459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.966439    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"]
Oct 30 19:39:47.002918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:46.980871    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h" podUID="55bc197f-e609-4a15-a323-194cc2fbbf95" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.120:50052\" within 1s: context deadline exceeded"
Oct 30 19:39:47.022742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.004157    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:39:47.022742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.011279    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.078782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.067893    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:39:47.078782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.068307    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded"
Oct 30 19:39:47.078782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.068773    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.078782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.069042    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:39:47.179714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.156722    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.215176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.215128    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.217278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.217248    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:47.328415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.328365    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.329463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.329429    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:39:47.397716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.394634    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:39:47.526787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.526672    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.587963 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.587921    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:47.960708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.949782    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.960708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.958088    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:47.971471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:47.971412    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:48.032713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.024774    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:48.126942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.126896    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:48.127621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.127510    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:48.128083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.128058    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:48.129805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.129765    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:48.141751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.141582    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:48.159715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.147310    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:39:48.177741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.161116    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:48.225661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.224985    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:39:48.226558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.225992    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:39:48.226763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.226737    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:39:48.227593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.227572    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:39:48.334852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.334802    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:48.385738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.336344    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:39:48.517676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.506997    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:48.531796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.531762    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:39:48.543768    4925 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.279s"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.543866    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.543895    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" event={"ID":"4386bb61-9200-4a7a-a3c4-ee73bdeb831a","Type":"ContainerStarted","Data":"c2f5c34a384dd131ddb66a5ab2a70a60da2fb8ee6ccad19e968871cb3cc1d88b"}
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.543947    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.544055    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.544086    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.544119    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.544147    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.544176    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.544203    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.544232    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:39:48.549014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.544261    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:39:48.595573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.595174    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:39:48.610331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.596259    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:39:48.610331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.607371    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:39:48.610331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.607814    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:39:48.610331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.608662    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:39:48.610331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.609136    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:39:48.613822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.613616    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:39:48.615120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.615096    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:39:48.864882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.864796    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"]
Oct 30 19:39:48.955774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.955740    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"]
Oct 30 19:39:48.987460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.985586    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:48.987460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.986888    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:48.988256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:48.988235    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:39:49.313304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:49.313210    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:49.637156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:49.633070    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" event={"ID":"4386bb61-9200-4a7a-a3c4-ee73bdeb831a","Type":"ContainerStarted","Data":"e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc"}
Oct 30 19:39:49.639649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:49.639266    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7" event={"ID":"c491d703-c607-45f6-958c-e3f1a942ed62","Type":"ContainerStarted","Data":"ac8a34a07fe11f8847201996fbdd7ecdac974622b064ef4c15bc6244e0a8d3fc"}
Oct 30 19:39:49.639649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:49.639321    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7" event={"ID":"c491d703-c607-45f6-958c-e3f1a942ed62","Type":"ContainerStarted","Data":"67cfa053a5f9b81ebccd77ac63ea05139c970bf972b9c8a4d85f983b210de213"}
Oct 30 19:39:49.661195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:49.661140    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" podStartSLOduration=8.66112561 podStartE2EDuration="8.66112561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:49.655729748 +0000 UTC m=+2024.558742785" watchObservedRunningTime="2026-10-30 19:39:49.66112561 +0000 UTC m=+2024.564138649"
Oct 30 19:39:49.685851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:49.685808    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7" podStartSLOduration=6.685794356 podStartE2EDuration="6.685794356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:49.679973001 +0000 UTC m=+2024.582986037" watchObservedRunningTime="2026-10-30 19:39:49.685794356 +0000 UTC m=+2024.588807392"
Oct 30 19:39:49.698845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:49.698809    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:50.304079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.289924    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:50.419629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.419602    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:39:50.457127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.457094    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"]
Oct 30 19:39:50.457900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.457881    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy"
Oct 30 19:39:50.457900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.457898    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy"
Oct 30 19:39:50.458080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.458068    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerName="istio-proxy"
Oct 30 19:39:50.462236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.462209    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.475559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.475330    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7e6199-openshift-default-dockercfg-zqwvn\""
Oct 30 19:39:50.484599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.484572    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"]
Oct 30 19:39:50.602272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602188    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-token\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-token\") pod \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") "
Oct 30 19:39:50.602449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602297    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istiod-ca-cert\") pod \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") "
Oct 30 19:39:50.602449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602326    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-podinfo\") pod \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") "
Oct 30 19:39:50.602449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602397    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-kube-api-access-rcr5q\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-kube-api-access-rcr5q\") pod \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") "
Oct 30 19:39:50.602637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602543    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-data\") pod \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") "
Oct 30 19:39:50.602637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602604    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-certs\") pod \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") "
Oct 30 19:39:50.602822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602649    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-socket\") pod \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") "
Oct 30 19:39:50.602822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602678    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-envoy\") pod \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") "
Oct 30 19:39:50.602822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602744    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istiod-ca-cert" pod "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" (UID: "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:50.602822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602757    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-credential-socket\") pod \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\" (UID: \"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9\") "
Oct 30 19:39:50.603054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602932    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-socket" pod "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" (UID: "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:50.603054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.602946    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-data" pod "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" (UID: "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:50.603170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.603154    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-certs" pod "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" (UID: "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:50.603292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.603257    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-credential-socket" pod "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" (UID: "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:50.603292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.603281    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-token\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.603500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.603350    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-socket\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.603567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.603542    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-credential-socket\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.604086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.603655    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb9m4\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-kube-api-access-pb9m4\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.604086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.603859    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-certs\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.604086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.604026    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7febe178-dee4-4add-9355-5da5c8dfc1ee-istiod-ca-cert\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.604333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.604207    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-envoy\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.604333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.604267    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-podinfo\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.604462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.604339    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-data\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.604524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.604477    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:50.604524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.604496    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:50.604524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.604512    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:50.604767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.604526    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:50.604767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.604541    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:50.605078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.605054    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-token" pod "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" (UID: "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:50.605407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.605380    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-envoy" pod "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" (UID: "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:50.605731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.605665    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-podinfo" pod "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" (UID: "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:50.605927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.605905    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-kube-api-access-rcr5q" pod "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" (UID: "c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9"). InnerVolumeSpecName "kube-api-access-rcr5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:50.649730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.649702    4925 generic.go:356] "Generic (PLEG): container finished" podID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" containerID="bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99" exitCode=0
Oct 30 19:39:50.649844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.649748    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"
Oct 30 19:39:50.649844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.649794    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" event={"ID":"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9","Type":"ContainerDied","Data":"bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99"}
Oct 30 19:39:50.649986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.649848    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr" event={"ID":"c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9","Type":"ContainerDied","Data":"7ba49453f2d91494dd99c952e48f1819109e60ff71ac171ce8d2aaec44105a21"}
Oct 30 19:39:50.649986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.649879    4925 scope.go:122] "RemoveContainer" containerID="bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99"
Oct 30 19:39:50.651790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.651738    4925 generic.go:356] "Generic (PLEG): container finished" podID="c491d703-c607-45f6-958c-e3f1a942ed62" containerID="ac8a34a07fe11f8847201996fbdd7ecdac974622b064ef4c15bc6244e0a8d3fc" exitCode=0
Oct 30 19:39:50.651899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.651820    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7" event={"ID":"c491d703-c607-45f6-958c-e3f1a942ed62","Type":"ContainerDied","Data":"ac8a34a07fe11f8847201996fbdd7ecdac974622b064ef4c15bc6244e0a8d3fc"}
Oct 30 19:39:50.662066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.662049    4925 scope.go:122] "RemoveContainer" containerID="bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99"
Oct 30 19:39:50.662337 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:39:50.662322    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99\": container with ID starting with bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99 not found: ID does not exist" containerID="bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99"
Oct 30 19:39:50.662397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.662346    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99"} err="failed to get container status \"bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99\": rpc error: code = NotFound desc = could not find container \"bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99\": container with ID starting with bfdcbccfe90b74655af08b1af92a8d04bc5b0b836bfb810212b3a39927c81e99 not found: ID does not exist"
Oct 30 19:39:50.694021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.693988    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"]
Oct 30 19:39:50.698750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.698723    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f2dbd9-openshift-default-776ff48945-s29qr"]
Oct 30 19:39:50.705773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.705748    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-envoy\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.705875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.705808    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-podinfo\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.705930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.705874    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-data\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706111    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-token\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706172    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-socket\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706221    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-credential-socket\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706228    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-data\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706271    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb9m4\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-kube-api-access-pb9m4\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706309    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-certs\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706382    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7febe178-dee4-4add-9355-5da5c8dfc1ee-istiod-ca-cert\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706491    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-credential-socket\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706515    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcr5q\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-kube-api-access-rcr5q\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:50.706607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706534    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:50.706607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706549    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:50.706607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706565    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:50.706897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706874    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-certs\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.706962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.706934    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-socket\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.707207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.707189    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7febe178-dee4-4add-9355-5da5c8dfc1ee-istiod-ca-cert\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.708962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.708944    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-envoy\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.709191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.709169    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-podinfo\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.718746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.718723    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-token\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.719104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.719079    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb9m4\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-kube-api-access-pb9m4\") pod \"e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") " pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:50.787965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:50.787936    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:51.086025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.085991    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"]
Oct 30 19:39:51.090331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.090300    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:51.092403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.092374    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35a893e-rgwsk"]
Oct 30 19:39:51.120626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.120507    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"]
Oct 30 19:39:51.121217 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:39:51.121188    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7febe178_dee4_4add_9355_5da5c8dfc1ee.slice/crio-633cf2766df82ee6f6efea4216baf7918dcfab74e258b09ad3fe3630a3e822db WatchSource:0}: Error finding container 633cf2766df82ee6f6efea4216baf7918dcfab74e258b09ad3fe3630a3e822db: Status 404 returned error can't find the container with id 633cf2766df82ee6f6efea4216baf7918dcfab74e258b09ad3fe3630a3e822db
Oct 30 19:39:51.125829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.124470    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:51.125829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.124562    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:51.125829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.124734    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:51.268153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.268120    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a059da2-0698-4630-bbba-b85e37e713f6" path="/var/lib/kubelet/pods/3a059da2-0698-4630-bbba-b85e37e713f6/volumes"
Oct 30 19:39:51.268828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.268807    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9" path="/var/lib/kubelet/pods/c9ce3ff9-3c3f-44ba-b6c5-5ad1e02140d9/volumes"
Oct 30 19:39:51.401958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.401924    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main" containerID="cri-o://8c62a7f7dc5cee89df964661d1379e07ca555ae97283a733ac1a002c4d6025f9" gracePeriod=45
Oct 30 19:39:51.666544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.666467    4925 generic.go:356] "Generic (PLEG): container finished" podID="9fb62679-e649-4990-93d7-745ec5d803b3" containerID="8c62a7f7dc5cee89df964661d1379e07ca555ae97283a733ac1a002c4d6025f9" exitCode=0
Oct 30 19:39:51.666715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.666553    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" event={"ID":"9fb62679-e649-4990-93d7-745ec5d803b3","Type":"ContainerDied","Data":"8c62a7f7dc5cee89df964661d1379e07ca555ae97283a733ac1a002c4d6025f9"}
Oct 30 19:39:51.668302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.668274    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" event={"ID":"7febe178-dee4-4add-9355-5da5c8dfc1ee","Type":"ContainerStarted","Data":"bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262"}
Oct 30 19:39:51.668417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.668321    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" event={"ID":"7febe178-dee4-4add-9355-5da5c8dfc1ee","Type":"ContainerStarted","Data":"633cf2766df82ee6f6efea4216baf7918dcfab74e258b09ad3fe3630a3e822db"}
Oct 30 19:39:51.699649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.699621    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:51.701587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.701545    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" podStartSLOduration=14.701532322 podStartE2EDuration="14.701532322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:51.695647564 +0000 UTC m=+2026.598660602" watchObservedRunningTime="2026-10-30 19:39:51.701532322 +0000 UTC m=+2026.604545359"
Oct 30 19:39:51.714885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.714862    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:39:51.717552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.717528    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:51.727752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.727730    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-tmp-dir\") pod \"9fb62679-e649-4990-93d7-745ec5d803b3\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") "
Oct 30 19:39:51.728278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.728257    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-tmp-dir" pod "9fb62679-e649-4990-93d7-745ec5d803b3" (UID: "9fb62679-e649-4990-93d7-745ec5d803b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:51.728479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.728461    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-home\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-home\") pod \"9fb62679-e649-4990-93d7-745ec5d803b3\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") "
Oct 30 19:39:51.728673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.728657    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-home" pod "9fb62679-e649-4990-93d7-745ec5d803b3" (UID: "9fb62679-e649-4990-93d7-745ec5d803b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:51.728785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.728766    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-dshm\") pod \"9fb62679-e649-4990-93d7-745ec5d803b3\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") "
Oct 30 19:39:51.729148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.729130    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb62679-e649-4990-93d7-745ec5d803b3-kube-api-access-qlvjk\" (UniqueName: \"kubernetes.io/projected/9fb62679-e649-4990-93d7-745ec5d803b3-kube-api-access-qlvjk\") pod \"9fb62679-e649-4990-93d7-745ec5d803b3\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") "
Oct 30 19:39:51.729541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.729526    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb62679-e649-4990-93d7-745ec5d803b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb62679-e649-4990-93d7-745ec5d803b3-tls-certs\") pod \"9fb62679-e649-4990-93d7-745ec5d803b3\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") "
Oct 30 19:39:51.729639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.729566    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-model-cache\") pod \"9fb62679-e649-4990-93d7-745ec5d803b3\" (UID: \"9fb62679-e649-4990-93d7-745ec5d803b3\") "
Oct 30 19:39:51.730061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.730041    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-model-cache" pod "9fb62679-e649-4990-93d7-745ec5d803b3" (UID: "9fb62679-e649-4990-93d7-745ec5d803b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:51.731414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.731395    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-dshm" pod "9fb62679-e649-4990-93d7-745ec5d803b3" (UID: "9fb62679-e649-4990-93d7-745ec5d803b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:51.731528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.731512    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb62679-e649-4990-93d7-745ec5d803b3-kube-api-access-qlvjk" pod "9fb62679-e649-4990-93d7-745ec5d803b3" (UID: "9fb62679-e649-4990-93d7-745ec5d803b3"). InnerVolumeSpecName "kube-api-access-qlvjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:51.732321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.732266    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb62679-e649-4990-93d7-745ec5d803b3-tls-certs" pod "9fb62679-e649-4990-93d7-745ec5d803b3" (UID: "9fb62679-e649-4990-93d7-745ec5d803b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:51.732427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.732402    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.732427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.732416    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.732553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.732438    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.732553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.732451    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb62679-e649-4990-93d7-745ec5d803b3-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.732553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.732459    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlvjk\" (UniqueName: \"kubernetes.io/projected/9fb62679-e649-4990-93d7-745ec5d803b3-kube-api-access-qlvjk\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.809887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.804792    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:51.811607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.811566    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:51.918767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.883292    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb62679-e649-4990-93d7-745ec5d803b3-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:51.965823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:51.965796    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:52.086925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.086887    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-xks-ca\") pod \"c491d703-c607-45f6-958c-e3f1a942ed62\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") "
Oct 30 19:39:52.087116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.087037    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-openshift-ca\") pod \"c491d703-c607-45f6-958c-e3f1a942ed62\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") "
Oct 30 19:39:52.087358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.087333    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-openshift-ca" pod "c491d703-c607-45f6-958c-e3f1a942ed62" (UID: "c491d703-c607-45f6-958c-e3f1a942ed62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:52.087431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.087395    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c491d703-c607-45f6-958c-e3f1a942ed62-kube-api-access-glcdp\" (UniqueName: \"kubernetes.io/projected/c491d703-c607-45f6-958c-e3f1a942ed62-kube-api-access-glcdp\") pod \"c491d703-c607-45f6-958c-e3f1a942ed62\" (UID: \"c491d703-c607-45f6-958c-e3f1a942ed62\") "
Oct 30 19:39:52.088281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.088250    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:52.089635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.089611    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-xks-ca" pod "c491d703-c607-45f6-958c-e3f1a942ed62" (UID: "c491d703-c607-45f6-958c-e3f1a942ed62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:52.095853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.095829    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c491d703-c607-45f6-958c-e3f1a942ed62-kube-api-access-glcdp" pod "c491d703-c607-45f6-958c-e3f1a942ed62" (UID: "c491d703-c607-45f6-958c-e3f1a942ed62"). InnerVolumeSpecName "kube-api-access-glcdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:52.151630 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:39:52.151593    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:39:52.191037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.190954    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glcdp\" (UniqueName: \"kubernetes.io/projected/c491d703-c607-45f6-958c-e3f1a942ed62-kube-api-access-glcdp\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:52.191037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.190989    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c491d703-c607-45f6-958c-e3f1a942ed62-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:52.700750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.694658    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:52.715445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.715415    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:39:52.717857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.717833    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"
Oct 30 19:39:52.717857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.717848    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp" event={"ID":"9fb62679-e649-4990-93d7-745ec5d803b3","Type":"ContainerDied","Data":"bfef3933995f922d194e28e35c08ea6a4132c6b84da2913f08b667a579455bb1"}
Oct 30 19:39:52.718045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.717899    4925 scope.go:122] "RemoveContainer" containerID="8c62a7f7dc5cee89df964661d1379e07ca555ae97283a733ac1a002c4d6025f9"
Oct 30 19:39:52.722108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.722088    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"
Oct 30 19:39:52.722234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.722113    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7" event={"ID":"c491d703-c607-45f6-958c-e3f1a942ed62","Type":"ContainerDied","Data":"67cfa053a5f9b81ebccd77ac63ea05139c970bf972b9c8a4d85f983b210de213"}
Oct 30 19:39:52.722234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.722161    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67cfa053a5f9b81ebccd77ac63ea05139c970bf972b9c8a4d85f983b210de213"
Oct 30 19:39:52.771925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.770217    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"]
Oct 30 19:39:52.778954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.778924    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e3586f/rate-test-model-e3586f-kserve-85c859cc7-zj9fp"]
Oct 30 19:39:52.795041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:52.795016    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:53.267104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.267070    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" path="/var/lib/kubelet/pods/9fb62679-e649-4990-93d7-745ec5d803b3/volumes"
Oct 30 19:39:53.536319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.536084    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:53.737170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.737137    4925 generic.go:356] "Generic (PLEG): container finished" podID="2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" containerID="e37adc3b266b78fb899fc98e600928a8120e659bdb3e35a94fb3e63813529c21" exitCode=0
Oct 30 19:39:53.737632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.737232    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz" event={"ID":"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac","Type":"ContainerDied","Data":"e37adc3b266b78fb899fc98e600928a8120e659bdb3e35a94fb3e63813529c21"}
Oct 30 19:39:53.739775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.739749    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:53.740988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.740966    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:39:53.772268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.772238    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:39:53.922020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.921987    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-kube-api-access-2tqbq\" (UniqueName: \"kubernetes.io/projected/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-kube-api-access-2tqbq\") pod \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\" (UID: \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\") "
Oct 30 19:39:53.922233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.922037    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-plugins-config-volume\") pod \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\" (UID: \"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac\") "
Oct 30 19:39:53.922456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.922431    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-plugins-config-volume" pod "2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" (UID: "2de63bc4-080b-4dc0-9729-f0e2b0fec1ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:53.923151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.923129    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:53.924517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:53.924494    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-kube-api-access-2tqbq" pod "2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" (UID: "2de63bc4-080b-4dc0-9729-f0e2b0fec1ac"). InnerVolumeSpecName "kube-api-access-2tqbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:54.000377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.000319    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:54.025015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.024967    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tqbq\" (UniqueName: \"kubernetes.io/projected/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac-kube-api-access-2tqbq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:54.167781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.167706    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:54.355780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.355150    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:54.617779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.617655    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:54.813076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.813046    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"
Oct 30 19:39:54.813834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.813280    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz" event={"ID":"2de63bc4-080b-4dc0-9729-f0e2b0fec1ac","Type":"ContainerDied","Data":"35c6c9de6427e5cc6c2b4251d4b269584e059854db511652ed15c403d8986c3e"}
Oct 30 19:39:54.813962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.813914    4925 scope.go:122] "RemoveContainer" containerID="e37adc3b266b78fb899fc98e600928a8120e659bdb3e35a94fb3e63813529c21"
Oct 30 19:39:54.841976 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:39:54.841945    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de63bc4_080b_4dc0_9729_f0e2b0fec1ac.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:39:54.852258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.852232    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"]
Oct 30 19:39:54.858846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:54.858823    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2dbd9-56d55cc99f-jhxpz"]
Oct 30 19:39:55.217401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.217362    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:55.225733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.225658    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"]
Oct 30 19:39:55.226514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.226496    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main"
Oct 30 19:39:55.226514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.226513    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main"
Oct 30 19:39:55.226671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.226533    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" containerName="payload-pre-processing"
Oct 30 19:39:55.226671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.226539    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" containerName="payload-pre-processing"
Oct 30 19:39:55.226671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.226572    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c491d703-c607-45f6-958c-e3f1a942ed62" containerName="revoke-keys"
Oct 30 19:39:55.226671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.226577    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c491d703-c607-45f6-958c-e3f1a942ed62" containerName="revoke-keys"
Oct 30 19:39:55.226944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.226709    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb62679-e649-4990-93d7-745ec5d803b3" containerName="main"
Oct 30 19:39:55.226944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.226719    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" containerName="payload-pre-processing"
Oct 30 19:39:55.226944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.226732    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c491d703-c607-45f6-958c-e3f1a942ed62" containerName="revoke-keys"
Oct 30 19:39:55.231716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.231671    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:39:55.238031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.238005    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e6199\""
Oct 30 19:39:55.243169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.243148    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"]
Oct 30 19:39:55.246000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.245965    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7e6199-dockercfg-xrq29\""
Oct 30 19:39:55.272758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.271838    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de63bc4-080b-4dc0-9729-f0e2b0fec1ac" path="/var/lib/kubelet/pods/2de63bc4-080b-4dc0-9729-f0e2b0fec1ac/volumes"
Oct 30 19:39:55.336119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.321412    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgvvs\" (UniqueName: \"kubernetes.io/projected/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-kube-api-access-cgvvs\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk\" (UID: \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:39:55.336119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.321662    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk\" (UID: \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:39:55.434714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.426072    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:55.436191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.436156    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgvvs\" (UniqueName: \"kubernetes.io/projected/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-kube-api-access-cgvvs\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk\" (UID: \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:39:55.436833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.436806    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk\" (UID: \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:39:55.440707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.437634    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk\" (UID: \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:39:55.468885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.468809    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgvvs\" (UniqueName: \"kubernetes.io/projected/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-kube-api-access-cgvvs\") pod \"payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk\" (UID: \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:39:55.579833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.579798    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:39:55.771380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:55.771348    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:56.118157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:56.117028    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"]
Oct 30 19:39:56.385053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:56.384961    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:56.734127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:56.733181    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:56.841877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:56.841840    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk" event={"ID":"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70","Type":"ContainerStarted","Data":"9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6"}
Oct 30 19:39:56.842063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:56.841904    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk" event={"ID":"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70","Type":"ContainerStarted","Data":"3e59c9584e05c41ab51529b1acb0c94519f4cc7bb076a43c24be0cdf10186a40"}
Oct 30 19:39:56.842063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:56.841938    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:39:56.860568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:56.860505    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk" podStartSLOduration=1.860487971 podStartE2EDuration="1.860487971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:56.859327852 +0000 UTC m=+2031.762340889" watchObservedRunningTime="2026-10-30 19:39:56.860487971 +0000 UTC m=+2031.763501013"
Oct 30 19:39:57.431714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:57.422157    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:57.888231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:57.888198    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:58.221715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:58.219838    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:59.069718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.065455    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:59.323815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.323737    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"]
Oct 30 19:39:59.345673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.345636    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"]
Oct 30 19:39:59.345944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.345856    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.350921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.350896    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:39:59.351108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.350929    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:59.409490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.409457    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3d7d02-p927m\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.409659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.409607    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3d7d02-p927m\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.409874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.409854    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdhg\" (UniqueName: \"kubernetes.io/projected/db391fde-68ad-4f37-97e6-de04690a45a9-kube-api-access-zgdhg\") pod \"maas-api-revoke-sub-8a3d7d02-p927m\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.511589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.511551    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdhg\" (UniqueName: \"kubernetes.io/projected/db391fde-68ad-4f37-97e6-de04690a45a9-kube-api-access-zgdhg\") pod \"maas-api-revoke-sub-8a3d7d02-p927m\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.511833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.511734    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3d7d02-p927m\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.511899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.511859    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3d7d02-p927m\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.512474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.512409    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3d7d02-p927m\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.514496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.514471    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3d7d02-p927m\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.523800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.523780    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdhg\" (UniqueName: \"kubernetes.io/projected/db391fde-68ad-4f37-97e6-de04690a45a9-kube-api-access-zgdhg\") pod \"maas-api-revoke-sub-8a3d7d02-p927m\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.670370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.670338    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:39:59.809620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.809560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:59.992509 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:39:59.992483    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb391fde_68ad_4f37_97e6_de04690a45a9.slice/crio-dd578dec63d7ccd9bf94fe500927045b02dbaad574364c4c59c463e42a5dadc7 WatchSource:0}: Error finding container dd578dec63d7ccd9bf94fe500927045b02dbaad574364c4c59c463e42a5dadc7: Status 404 returned error can't find the container with id dd578dec63d7ccd9bf94fe500927045b02dbaad574364c4c59c463e42a5dadc7
Oct 30 19:39:59.994464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:39:59.994431    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"]
Oct 30 19:40:00.384865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:00.384824    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:00.551715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:00.547354    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"]
Oct 30 19:40:00.611878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:00.611629    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:00.627713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:00.624276    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1fa500-xv8g7"]
Oct 30 19:40:01.030727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:01.025363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:01.092708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:01.090225    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m" event={"ID":"db391fde-68ad-4f37-97e6-de04690a45a9","Type":"ContainerStarted","Data":"b18ed36f78f16ae68ddb1a34f6afb084ce4518387c9383efc8b6d2a15ae21020"}
Oct 30 19:40:01.092708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:01.090305    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m" event={"ID":"db391fde-68ad-4f37-97e6-de04690a45a9","Type":"ContainerStarted","Data":"dd578dec63d7ccd9bf94fe500927045b02dbaad574364c4c59c463e42a5dadc7"}
Oct 30 19:40:01.120296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:01.120239    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m" podStartSLOduration=2.120225193 podStartE2EDuration="2.120225193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:01.114672924 +0000 UTC m=+2036.017685956" watchObservedRunningTime="2026-10-30 19:40:01.120225193 +0000 UTC m=+2036.023238230"
Oct 30 19:40:01.232459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:01.232409    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:01.268138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:01.268109    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c491d703-c607-45f6-958c-e3f1a942ed62" path="/var/lib/kubelet/pods/c491d703-c607-45f6-958c-e3f1a942ed62/volumes"
Oct 30 19:40:01.909671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:01.909625    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:02.099816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:02.099774    4925 generic.go:356] "Generic (PLEG): container finished" podID="db391fde-68ad-4f37-97e6-de04690a45a9" containerID="b18ed36f78f16ae68ddb1a34f6afb084ce4518387c9383efc8b6d2a15ae21020" exitCode=0
Oct 30 19:40:02.100014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:02.099861    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m" event={"ID":"db391fde-68ad-4f37-97e6-de04690a45a9","Type":"ContainerDied","Data":"b18ed36f78f16ae68ddb1a34f6afb084ce4518387c9383efc8b6d2a15ae21020"}
Oct 30 19:40:02.859734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:02.859681    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:03.312415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.312387    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:40:03.450949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.450915    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db391fde-68ad-4f37-97e6-de04690a45a9-kube-api-access-zgdhg\" (UniqueName: \"kubernetes.io/projected/db391fde-68ad-4f37-97e6-de04690a45a9-kube-api-access-zgdhg\") pod \"db391fde-68ad-4f37-97e6-de04690a45a9\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") "
Oct 30 19:40:03.451124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.451056    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-openshift-ca\") pod \"db391fde-68ad-4f37-97e6-de04690a45a9\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") "
Oct 30 19:40:03.451229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.451211    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-xks-ca\") pod \"db391fde-68ad-4f37-97e6-de04690a45a9\" (UID: \"db391fde-68ad-4f37-97e6-de04690a45a9\") "
Oct 30 19:40:03.451468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.451447    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-openshift-ca" pod "db391fde-68ad-4f37-97e6-de04690a45a9" (UID: "db391fde-68ad-4f37-97e6-de04690a45a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:03.452041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.452019    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:03.453388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.453359    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-xks-ca" pod "db391fde-68ad-4f37-97e6-de04690a45a9" (UID: "db391fde-68ad-4f37-97e6-de04690a45a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:03.453482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.453450    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db391fde-68ad-4f37-97e6-de04690a45a9-kube-api-access-zgdhg" pod "db391fde-68ad-4f37-97e6-de04690a45a9" (UID: "db391fde-68ad-4f37-97e6-de04690a45a9"). InnerVolumeSpecName "kube-api-access-zgdhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:03.554458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.554416    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db391fde-68ad-4f37-97e6-de04690a45a9-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:03.554458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:03.554462    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdhg\" (UniqueName: \"kubernetes.io/projected/db391fde-68ad-4f37-97e6-de04690a45a9-kube-api-access-zgdhg\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.075596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:04.075550    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:04.121281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:04.121251    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m" event={"ID":"db391fde-68ad-4f37-97e6-de04690a45a9","Type":"ContainerDied","Data":"dd578dec63d7ccd9bf94fe500927045b02dbaad574364c4c59c463e42a5dadc7"}
Oct 30 19:40:04.121479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:04.121302    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd578dec63d7ccd9bf94fe500927045b02dbaad574364c4c59c463e42a5dadc7"
Oct 30 19:40:04.121479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:04.121256    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"
Oct 30 19:40:04.897352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:04.897319    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:05.792713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:05.790843    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:06.708579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:06.708541    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:07.097043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:07.096953    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"]
Oct 30 19:40:07.097237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:07.097199    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy" containerID="cri-o://3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8" gracePeriod=30
Oct 30 19:40:07.541822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:07.541784    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:07.854735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:07.854645    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:40:08.203408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:08.203371    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:08.983058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:08.982979    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:09.187290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:09.186586    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:09.418598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:09.418558    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"]
Oct 30 19:40:09.418862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:09.418836    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy" containerID="cri-o://7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40" gracePeriod=30
Oct 30 19:40:09.571182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:09.571145    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:09.910767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:09.910736    4925 scope.go:122] "RemoveContainer" containerID="f1a4fa788a0bf531d508ff6455427b2e0ce37e23530f43c417c8acf8808c51f0"
Oct 30 19:40:09.927995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:09.927951    4925 scope.go:122] "RemoveContainer" containerID="b0b36489436a6d7413ecfc19dbae4811705aecddd0d491b71103684bb599c263"
Oct 30 19:40:10.090736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.090371    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:10.592935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.592899    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"]
Oct 30 19:40:10.594039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.594017    4925 cpu_manager.go:414] "containerMap: removing container" podUID="db391fde-68ad-4f37-97e6-de04690a45a9" containerName="revoke-keys"
Oct 30 19:40:10.594039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.594038    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db391fde-68ad-4f37-97e6-de04690a45a9" containerName="revoke-keys"
Oct 30 19:40:10.594265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.594235    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="db391fde-68ad-4f37-97e6-de04690a45a9" containerName="revoke-keys"
Oct 30 19:40:10.600658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.600637    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.605066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.605041    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"]
Oct 30 19:40:10.605534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.605513    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:10.606946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.606922    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:40:10.637215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.637182    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04830de-ec49-44a4-9de6-78743d480279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf9544c0-m99lt\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.637395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.637344    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6b5b\" (UniqueName: \"kubernetes.io/projected/a04830de-ec49-44a4-9de6-78743d480279-kube-api-access-v6b5b\") pod \"maas-api-revoke-sub-cf9544c0-m99lt\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.637541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.637522    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04830de-ec49-44a4-9de6-78743d480279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf9544c0-m99lt\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.738553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.738519    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04830de-ec49-44a4-9de6-78743d480279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf9544c0-m99lt\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.738797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.738670    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04830de-ec49-44a4-9de6-78743d480279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf9544c0-m99lt\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.738797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.738777    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6b5b\" (UniqueName: \"kubernetes.io/projected/a04830de-ec49-44a4-9de6-78743d480279-kube-api-access-v6b5b\") pod \"maas-api-revoke-sub-cf9544c0-m99lt\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.739222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.739201    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04830de-ec49-44a4-9de6-78743d480279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf9544c0-m99lt\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.741306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.741288    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04830de-ec49-44a4-9de6-78743d480279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf9544c0-m99lt\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.751022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.751000    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6b5b\" (UniqueName: \"kubernetes.io/projected/a04830de-ec49-44a4-9de6-78743d480279-kube-api-access-v6b5b\") pod \"maas-api-revoke-sub-cf9544c0-m99lt\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:10.921131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:10.921104    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:11.069250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.062166    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:11.188993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.188880    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"]
Oct 30 19:40:11.199614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.196647    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.206051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.205629    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b281d5\"/\"openshift-service-ca.crt\""
Oct 30 19:40:11.211815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.211758    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"]
Oct 30 19:40:11.216056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.216033    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b281d5\"/\"auth-test-model-b281d5-kserve-self-signed-certs\""
Oct 30 19:40:11.218117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.216254    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b281d5\"/\"kube-root-ca.crt\""
Oct 30 19:40:11.218117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.216330    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b281d5\"/\"default-dockercfg-5hl6f\""
Oct 30 19:40:11.235055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.233904    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"]
Oct 30 19:40:11.235055 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:40:11.234265    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04830de_ec49_44a4_9de6_78743d480279.slice/crio-fe996cc1671eb4fab74b81669b22aa9585397fbc76c97aff4a750a2bdf2a7b31 WatchSource:0}: Error finding container fe996cc1671eb4fab74b81669b22aa9585397fbc76c97aff4a750a2bdf2a7b31: Status 404 returned error can't find the container with id fe996cc1671eb4fab74b81669b22aa9585397fbc76c97aff4a750a2bdf2a7b31
Oct 30 19:40:11.293794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.293761    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:11.320360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.320151    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grk6x\" (UniqueName: \"kubernetes.io/projected/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-kube-api-access-grk6x\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.320360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.320280    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-home\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.320602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.320434    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tmp-dir\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.320663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.320598    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tls-certs\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.320777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.320740    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-model-cache\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.320850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.320828    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-dshm\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.429714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.424217    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grk6x\" (UniqueName: \"kubernetes.io/projected/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-kube-api-access-grk6x\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.429714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.424315    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-home\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.429714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.424399    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tmp-dir\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.429714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.424536    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tls-certs\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.429714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.424615    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-model-cache\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.437712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.435954    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-model-cache\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.437712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.436058    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-dshm\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.452778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.451955    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-dshm\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.452778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.452589    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-home\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.453006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.452869    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tmp-dir\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.461243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.455864    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tls-certs\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.484940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.484908    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grk6x\" (UniqueName: \"kubernetes.io/projected/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-kube-api-access-grk6x\") pod \"auth-test-model-b281d5-kserve-7f968df7d9-msljt\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") " pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.527036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.526992    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:11.878981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.878947    4925 patch_prober.go:29] interesting pod/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:40:11.879156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.879018    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:40:11.940067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.939885    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:11.982711 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:40:11.982659    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5f698e5_7f5d_4d87_a007_c43494eb1aa4.slice/crio-d7e9cbe20ad09dcd8f86122f04a09229852286644a496684d89bc91966bb4b79 WatchSource:0}: Error finding container d7e9cbe20ad09dcd8f86122f04a09229852286644a496684d89bc91966bb4b79: Status 404 returned error can't find the container with id d7e9cbe20ad09dcd8f86122f04a09229852286644a496684d89bc91966bb4b79
Oct 30 19:40:11.985225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:11.984959    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"]
Oct 30 19:40:12.205428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.205391    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt" event={"ID":"a04830de-ec49-44a4-9de6-78743d480279","Type":"ContainerStarted","Data":"d35ba3d709e2e86847e08dcf3b023d975f86673246494d84d9d800e0e5eeaa72"}
Oct 30 19:40:12.205872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.205458    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt" event={"ID":"a04830de-ec49-44a4-9de6-78743d480279","Type":"ContainerStarted","Data":"fe996cc1671eb4fab74b81669b22aa9585397fbc76c97aff4a750a2bdf2a7b31"}
Oct 30 19:40:12.207321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.207295    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" event={"ID":"e5f698e5-7f5d-4d87-a007-c43494eb1aa4","Type":"ContainerStarted","Data":"5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429"}
Oct 30 19:40:12.207436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.207333    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" event={"ID":"e5f698e5-7f5d-4d87-a007-c43494eb1aa4","Type":"ContainerStarted","Data":"d7e9cbe20ad09dcd8f86122f04a09229852286644a496684d89bc91966bb4b79"}
Oct 30 19:40:12.221816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.221581    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt" podStartSLOduration=2.2215653140000002 podStartE2EDuration="2.221565314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:12.221128762 +0000 UTC m=+2047.124141800" watchObservedRunningTime="2026-10-30 19:40:12.221565314 +0000 UTC m=+2047.124578352"
Oct 30 19:40:12.262047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.255218    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" podStartSLOduration=1.255197396 podStartE2EDuration="1.255197396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:12.246545509 +0000 UTC m=+2047.149558549" watchObservedRunningTime="2026-10-30 19:40:12.255197396 +0000 UTC m=+2047.158210434"
Oct 30 19:40:12.516724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.516615    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:40:12.528397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.528371    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:12.560540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.560510    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"]
Oct 30 19:40:12.561433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.561408    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy"
Oct 30 19:40:12.561624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.561596    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy"
Oct 30 19:40:12.561913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.561890    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6792554-663c-4785-940c-da31095bd57f" containerName="istio-proxy"
Oct 30 19:40:12.566217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.566197    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.581933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.581906    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"]
Oct 30 19:40:12.587048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.587023    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-socket\") pod \"f6792554-663c-4785-940c-da31095bd57f\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") "
Oct 30 19:40:12.587187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.587133    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-istio-token\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-istio-token\") pod \"f6792554-663c-4785-940c-da31095bd57f\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") "
Oct 30 19:40:12.587187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.587165    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-credential-socket\") pod \"f6792554-663c-4785-940c-da31095bd57f\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") "
Oct 30 19:40:12.587296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.587261    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-socket" pod "f6792554-663c-4785-940c-da31095bd57f" (UID: "f6792554-663c-4785-940c-da31095bd57f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:12.587522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.587505    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-credential-socket" pod "f6792554-663c-4785-940c-da31095bd57f" (UID: "f6792554-663c-4785-940c-da31095bd57f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:12.587739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.587665    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6792554-663c-4785-940c-da31095bd57f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6792554-663c-4785-940c-da31095bd57f-istio-podinfo\") pod \"f6792554-663c-4785-940c-da31095bd57f\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") "
Oct 30 19:40:12.588388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.588368    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6792554-663c-4785-940c-da31095bd57f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6792554-663c-4785-940c-da31095bd57f-istiod-ca-cert\") pod \"f6792554-663c-4785-940c-da31095bd57f\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") "
Oct 30 19:40:12.588505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.588412    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-data\") pod \"f6792554-663c-4785-940c-da31095bd57f\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") "
Oct 30 19:40:12.588505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.588472    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-kube-api-access-565tw\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-kube-api-access-565tw\") pod \"f6792554-663c-4785-940c-da31095bd57f\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") "
Oct 30 19:40:12.588620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.588501    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-envoy\") pod \"f6792554-663c-4785-940c-da31095bd57f\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") "
Oct 30 19:40:12.588620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.588555    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-certs\") pod \"f6792554-663c-4785-940c-da31095bd57f\" (UID: \"f6792554-663c-4785-940c-da31095bd57f\") "
Oct 30 19:40:12.589046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.589027    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-data" pod "f6792554-663c-4785-940c-da31095bd57f" (UID: "f6792554-663c-4785-940c-da31095bd57f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:12.589440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.589419    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6792554-663c-4785-940c-da31095bd57f-istiod-ca-cert" pod "f6792554-663c-4785-940c-da31095bd57f" (UID: "f6792554-663c-4785-940c-da31095bd57f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:12.589912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.589874    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-certs" pod "f6792554-663c-4785-940c-da31095bd57f" (UID: "f6792554-663c-4785-940c-da31095bd57f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:12.591023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.590915    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6792554-663c-4785-940c-da31095bd57f-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:12.591023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.590946    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:12.591023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.590961    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:12.591023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.590976    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:12.591023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.590991    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:12.591338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.591167    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-istio-token" pod "f6792554-663c-4785-940c-da31095bd57f" (UID: "f6792554-663c-4785-940c-da31095bd57f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:12.592130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.592103    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6792554-663c-4785-940c-da31095bd57f-istio-podinfo" pod "f6792554-663c-4785-940c-da31095bd57f" (UID: "f6792554-663c-4785-940c-da31095bd57f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:40:12.592600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.592570    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-envoy" pod "f6792554-663c-4785-940c-da31095bd57f" (UID: "f6792554-663c-4785-940c-da31095bd57f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:12.595187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.593404    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-kube-api-access-565tw" pod "f6792554-663c-4785-940c-da31095bd57f" (UID: "f6792554-663c-4785-940c-da31095bd57f"). InnerVolumeSpecName "kube-api-access-565tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:12.693047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693008    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.693213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693156    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.693213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693190    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.693335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693237    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz68n\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-kube-api-access-wz68n\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.693486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693455    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdffc51-409a-4865-9be7-10e8d20e0882-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.693572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693563    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.693682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693597    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.693784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693737    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.693846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693816    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.693922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693910    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:12.693977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693924    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6792554-663c-4785-940c-da31095bd57f-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:12.693977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693934    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-565tw\" (UniqueName: \"kubernetes.io/projected/f6792554-663c-4785-940c-da31095bd57f-kube-api-access-565tw\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:12.693977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.693944    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6792554-663c-4785-940c-da31095bd57f-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:12.795603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.795528    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.795787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.795622    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.795787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.795662    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.795916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.795787    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.795916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.795813    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.795916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.795842    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz68n\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-kube-api-access-wz68n\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.796090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.795946    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdffc51-409a-4865-9be7-10e8d20e0882-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.796090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.796026    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.796090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.796061    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.796236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.796136    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.796499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.796476    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.796677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.796640    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdffc51-409a-4865-9be7-10e8d20e0882-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.796887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.796864    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.796965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.796931    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.798150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.798130    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.798531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.798513    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.800821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.800799    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:12.807398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.807376    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.807743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.807728    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz68n\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-kube-api-access-wz68n\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:12.891724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:12.890067    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:13.017625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.017593    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:13.220643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.220602    4925 generic.go:356] "Generic (PLEG): container finished" podID="f6792554-663c-4785-940c-da31095bd57f" containerID="3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8" exitCode=0
Oct 30 19:40:13.221172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.220747    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" event={"ID":"f6792554-663c-4785-940c-da31095bd57f","Type":"ContainerDied","Data":"3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8"}
Oct 30 19:40:13.221172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.220806    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r" event={"ID":"f6792554-663c-4785-940c-da31095bd57f","Type":"ContainerDied","Data":"0ef509ac95fb476f0ccea07a7fe623428c7ace0c3325eebe1bc7a92ecfa15156"}
Oct 30 19:40:13.221172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.220837    4925 scope.go:122] "RemoveContainer" containerID="3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8"
Oct 30 19:40:13.221172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.220991    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"
Oct 30 19:40:13.227299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.227267    4925 generic.go:356] "Generic (PLEG): container finished" podID="a04830de-ec49-44a4-9de6-78743d480279" containerID="d35ba3d709e2e86847e08dcf3b023d975f86673246494d84d9d800e0e5eeaa72" exitCode=0
Oct 30 19:40:13.231644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.228880    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt" event={"ID":"a04830de-ec49-44a4-9de6-78743d480279","Type":"ContainerDied","Data":"d35ba3d709e2e86847e08dcf3b023d975f86673246494d84d9d800e0e5eeaa72"}
Oct 30 19:40:13.261933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.261902    4925 scope.go:122] "RemoveContainer" containerID="3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8"
Oct 30 19:40:13.262330 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:40:13.262304    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8\": container with ID starting with 3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8 not found: ID does not exist" containerID="3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8"
Oct 30 19:40:13.262437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.262343    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8"} err="failed to get container status \"3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8\": rpc error: code = NotFound desc = could not find container \"3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8\": container with ID starting with 3bc0ef57fba24a5adc8349d1e2ec6f64e28377f281d30114adad8f28e50663e8 not found: ID does not exist"
Oct 30 19:40:13.288209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.288177    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"]
Oct 30 19:40:13.295711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.295660    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-vbs5r"]
Oct 30 19:40:13.438423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.434346    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"]
Oct 30 19:40:13.449794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.448960    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:13.449794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.449060    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:13.449794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.449100    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:13.790627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:13.790586    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:14.255501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.255457    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" event={"ID":"2cdffc51-409a-4865-9be7-10e8d20e0882","Type":"ContainerStarted","Data":"a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546"}
Oct 30 19:40:14.256078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.255527    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" event={"ID":"2cdffc51-409a-4865-9be7-10e8d20e0882","Type":"ContainerStarted","Data":"078c5e1ce7b21fcab8e78bd03b790c87e7eab50e834ce7c27c6f11b54408b4c8"}
Oct 30 19:40:14.282560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.282491    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podStartSLOduration=89.282470832 podStartE2EDuration="1m29.282470832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:14.278738686 +0000 UTC m=+2049.181751718" watchObservedRunningTime="2026-10-30 19:40:14.282470832 +0000 UTC m=+2049.185483927"
Oct 30 19:40:14.544508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.544477    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:14.677845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.677810    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:14.737083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.737046    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a04830de-ec49-44a4-9de6-78743d480279-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04830de-ec49-44a4-9de6-78743d480279-maas-api-xks-ca\") pod \"a04830de-ec49-44a4-9de6-78743d480279\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") "
Oct 30 19:40:14.741649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.740605    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a04830de-ec49-44a4-9de6-78743d480279-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04830de-ec49-44a4-9de6-78743d480279-maas-api-openshift-ca\") pod \"a04830de-ec49-44a4-9de6-78743d480279\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") "
Oct 30 19:40:14.745982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.745022    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04830de-ec49-44a4-9de6-78743d480279-kube-api-access-v6b5b\" (UniqueName: \"kubernetes.io/projected/a04830de-ec49-44a4-9de6-78743d480279-kube-api-access-v6b5b\") pod \"a04830de-ec49-44a4-9de6-78743d480279\" (UID: \"a04830de-ec49-44a4-9de6-78743d480279\") "
Oct 30 19:40:14.749978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.749947    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04830de-ec49-44a4-9de6-78743d480279-maas-api-openshift-ca" pod "a04830de-ec49-44a4-9de6-78743d480279" (UID: "a04830de-ec49-44a4-9de6-78743d480279"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:14.754556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.754529    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a04830de-ec49-44a4-9de6-78743d480279-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:14.765536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.765495    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a04830de-ec49-44a4-9de6-78743d480279-maas-api-xks-ca" pod "a04830de-ec49-44a4-9de6-78743d480279" (UID: "a04830de-ec49-44a4-9de6-78743d480279"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:14.783131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.772304    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04830de-ec49-44a4-9de6-78743d480279-kube-api-access-v6b5b" pod "a04830de-ec49-44a4-9de6-78743d480279" (UID: "a04830de-ec49-44a4-9de6-78743d480279"). InnerVolumeSpecName "kube-api-access-v6b5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:14.873953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.870140    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6b5b\" (UniqueName: \"kubernetes.io/projected/a04830de-ec49-44a4-9de6-78743d480279-kube-api-access-v6b5b\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:14.873953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.870180    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a04830de-ec49-44a4-9de6-78743d480279-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:14.920786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.919784    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:14.948082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.945848    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:40:14.948082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:14.945911    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 19:40:15.071173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.071130    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:40:15.139766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.138956    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"]
Oct 30 19:40:15.143325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.140186    4925 cpu_manager.go:414] "containerMap: removing container" podUID="a04830de-ec49-44a4-9de6-78743d480279" containerName="revoke-keys"
Oct 30 19:40:15.143325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.140211    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04830de-ec49-44a4-9de6-78743d480279" containerName="revoke-keys"
Oct 30 19:40:15.143325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.140248    4925 cpu_manager.go:414] "containerMap: removing container" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy"
Oct 30 19:40:15.143325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.140256    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy"
Oct 30 19:40:15.143325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.140473    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04830de-ec49-44a4-9de6-78743d480279" containerName="revoke-keys"
Oct 30 19:40:15.143325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.140488    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" containerName="istio-proxy"
Oct 30 19:40:15.144920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.144900    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"]
Oct 30 19:40:15.145021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.144998    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.182536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.182510    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/942788f9-4ccc-4fec-917a-3ff94573071c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942788f9-4ccc-4fec-917a-3ff94573071c-istio-podinfo\") pod \"942788f9-4ccc-4fec-917a-3ff94573071c\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") "
Oct 30 19:40:15.182650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.182555    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-credential-socket\") pod \"942788f9-4ccc-4fec-917a-3ff94573071c\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") "
Oct 30 19:40:15.182650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.182623    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942788f9-4ccc-4fec-917a-3ff94573071c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942788f9-4ccc-4fec-917a-3ff94573071c-istiod-ca-cert\") pod \"942788f9-4ccc-4fec-917a-3ff94573071c\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") "
Oct 30 19:40:15.182763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.182659    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-data\") pod \"942788f9-4ccc-4fec-917a-3ff94573071c\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") "
Oct 30 19:40:15.182763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.182744    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-istio-token\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-istio-token\") pod \"942788f9-4ccc-4fec-917a-3ff94573071c\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") "
Oct 30 19:40:15.182825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.182769    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-certs\") pod \"942788f9-4ccc-4fec-917a-3ff94573071c\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") "
Oct 30 19:40:15.182880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.182866    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-socket\") pod \"942788f9-4ccc-4fec-917a-3ff94573071c\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") "
Oct 30 19:40:15.182990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.182978    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-envoy\") pod \"942788f9-4ccc-4fec-917a-3ff94573071c\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") "
Oct 30 19:40:15.183054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183018    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-kube-api-access-82pcl\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-kube-api-access-82pcl\") pod \"942788f9-4ccc-4fec-917a-3ff94573071c\" (UID: \"942788f9-4ccc-4fec-917a-3ff94573071c\") "
Oct 30 19:40:15.183243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183224    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.183317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183257    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.183317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183281    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.183427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183350    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-socket" pod "942788f9-4ccc-4fec-917a-3ff94573071c" (UID: "942788f9-4ccc-4fec-917a-3ff94573071c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:15.183427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183358    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942788f9-4ccc-4fec-917a-3ff94573071c-istiod-ca-cert" pod "942788f9-4ccc-4fec-917a-3ff94573071c" (UID: "942788f9-4ccc-4fec-917a-3ff94573071c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:15.183427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183367    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44v4\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-kube-api-access-q44v4\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.183578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183558    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.183710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183658    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.183845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183819    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-certs" pod "942788f9-4ccc-4fec-917a-3ff94573071c" (UID: "942788f9-4ccc-4fec-917a-3ff94573071c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:15.183957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.183932    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dc8df8-e90f-4697-aaa1-a332423097d2-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.184169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.184150    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.184295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.184271    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-data" pod "942788f9-4ccc-4fec-917a-3ff94573071c" (UID: "942788f9-4ccc-4fec-917a-3ff94573071c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:15.184410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.184388    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.184478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.184393    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-credential-socket" pod "942788f9-4ccc-4fec-917a-3ff94573071c" (UID: "942788f9-4ccc-4fec-917a-3ff94573071c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:15.184591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.184566    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.184721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.184599    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942788f9-4ccc-4fec-917a-3ff94573071c-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.184721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.184614    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.184721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.184627    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.184721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.184642    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.186394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.186368    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-envoy" pod "942788f9-4ccc-4fec-917a-3ff94573071c" (UID: "942788f9-4ccc-4fec-917a-3ff94573071c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:15.186793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.186770    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/942788f9-4ccc-4fec-917a-3ff94573071c-istio-podinfo" pod "942788f9-4ccc-4fec-917a-3ff94573071c" (UID: "942788f9-4ccc-4fec-917a-3ff94573071c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:40:15.187242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.187215    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-kube-api-access-82pcl" pod "942788f9-4ccc-4fec-917a-3ff94573071c" (UID: "942788f9-4ccc-4fec-917a-3ff94573071c"). InnerVolumeSpecName "kube-api-access-82pcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:15.187340 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.187322    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-istio-token" pod "942788f9-4ccc-4fec-917a-3ff94573071c" (UID: "942788f9-4ccc-4fec-917a-3ff94573071c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:15.265679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.265644    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6792554-663c-4785-940c-da31095bd57f" path="/var/lib/kubelet/pods/f6792554-663c-4785-940c-da31095bd57f/volumes"
Oct 30 19:40:15.266449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.266424    4925 generic.go:356] "Generic (PLEG): container finished" podID="942788f9-4ccc-4fec-917a-3ff94573071c" containerID="7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40" exitCode=0
Oct 30 19:40:15.266603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.266585    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"
Oct 30 19:40:15.266720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.266665    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" event={"ID":"942788f9-4ccc-4fec-917a-3ff94573071c","Type":"ContainerDied","Data":"7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40"}
Oct 30 19:40:15.266802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.266766    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz" event={"ID":"942788f9-4ccc-4fec-917a-3ff94573071c","Type":"ContainerDied","Data":"dabe85d8bdfe5c8f8a5c5f23cac4efedd3cdcb518141d026085667da498d93c6"}
Oct 30 19:40:15.266858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.266804    4925 scope.go:122] "RemoveContainer" containerID="7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40"
Oct 30 19:40:15.268956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.268934    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"
Oct 30 19:40:15.269117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.268948    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt" event={"ID":"a04830de-ec49-44a4-9de6-78743d480279","Type":"ContainerDied","Data":"fe996cc1671eb4fab74b81669b22aa9585397fbc76c97aff4a750a2bdf2a7b31"}
Oct 30 19:40:15.269246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.269233    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe996cc1671eb4fab74b81669b22aa9585397fbc76c97aff4a750a2bdf2a7b31"
Oct 30 19:40:15.284103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.284085    4925 scope.go:122] "RemoveContainer" containerID="7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40"
Oct 30 19:40:15.284379 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:40:15.284355    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40\": container with ID starting with 7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40 not found: ID does not exist" containerID="7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40"
Oct 30 19:40:15.284436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.284393    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40"} err="failed to get container status \"7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40\": rpc error: code = NotFound desc = could not find container \"7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40\": container with ID starting with 7ac04534e3cc86bd96aae5a3421f59e6837a1f3b90d2b294f2eea74e493cfa40 not found: ID does not exist"
Oct 30 19:40:15.285625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.285595    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.285804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.285787    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.285892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.285833    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.285892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.285866    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.286009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.285898    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.286009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286001    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44v4\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-kube-api-access-q44v4\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.286116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286013    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-data\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.286116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286105    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.286223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286170    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.286223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286215    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dc8df8-e90f-4697-aaa1-a332423097d2-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.286345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286286    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-certs\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.286345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286292    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.286345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286339    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942788f9-4ccc-4fec-917a-3ff94573071c-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.286514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286352    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.286514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286366    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942788f9-4ccc-4fec-917a-3ff94573071c-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.286514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286382    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82pcl\" (UniqueName: \"kubernetes.io/projected/942788f9-4ccc-4fec-917a-3ff94573071c-kube-api-access-82pcl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:15.286784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.286645    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-credential-socket\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.287227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.287204    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dc8df8-e90f-4697-aaa1-a332423097d2-istiod-ca-cert\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.289387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.289365    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-envoy\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.290458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.290404    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-podinfo\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.326307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.326274    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"]
Oct 30 19:40:15.327516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.327485    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-token\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.328850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.328825    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44v4\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-kube-api-access-q44v4\") pod \"e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") " pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.331854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.331828    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1fa500-openshift-default-5f684fd55f-h7zzz"]
Oct 30 19:40:15.425263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.425053    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:15.474464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.474138    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:15.828008 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:40:15.827978    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94dc8df8_e90f_4697_aaa1_a332423097d2.slice/crio-a50cdf149925be318acf0ca3551827d747851c2baad3ae42ed9c3416b084efb0 WatchSource:0}: Error finding container a50cdf149925be318acf0ca3551827d747851c2baad3ae42ed9c3416b084efb0: Status 404 returned error can't find the container with id a50cdf149925be318acf0ca3551827d747851c2baad3ae42ed9c3416b084efb0
Oct 30 19:40:15.828500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.828477    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"]
Oct 30 19:40:15.831611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.831576    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:15.831761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.831653    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:15.831761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.831721    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:15.890757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.890726    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:40:15.890865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:15.890786    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 19:40:16.321441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:16.321397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:16.402826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:16.396424    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" event={"ID":"94dc8df8-e90f-4697-aaa1-a332423097d2","Type":"ContainerStarted","Data":"e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902"}
Oct 30 19:40:16.402826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:16.396480    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" event={"ID":"94dc8df8-e90f-4697-aaa1-a332423097d2","Type":"ContainerStarted","Data":"a50cdf149925be318acf0ca3551827d747851c2baad3ae42ed9c3416b084efb0"}
Oct 30 19:40:16.449438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:16.449353    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podStartSLOduration=91.449336941 podStartE2EDuration="1m31.449336941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:16.443744802 +0000 UTC m=+2051.346757838" watchObservedRunningTime="2026-10-30 19:40:16.449336941 +0000 UTC m=+2051.352349978"
Oct 30 19:40:16.475783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:16.475755    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:16.477456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:16.477420    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:40:16.477673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:16.477477    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:40:16.890996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:16.890966    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:40:16.891187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:16.891016    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 19:40:17.102714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:17.102439    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:17.275764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:17.274656    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942788f9-4ccc-4fec-917a-3ff94573071c" path="/var/lib/kubelet/pods/942788f9-4ccc-4fec-917a-3ff94573071c/volumes"
Oct 30 19:40:17.476169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:17.476130    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:40:17.476551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:17.476200    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:40:17.808710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:17.808608    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:17.895715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:17.894785    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:40:17.895715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:17.894846    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 19:40:18.475310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:18.475274    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:40:18.475551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:18.475337    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:40:18.691717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:18.689228    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:18.890723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:18.890669    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:40:18.890884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:18.890737    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 19:40:19.376773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:19.375137    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:19.477818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:19.477266    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:40:19.477818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:19.477327    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:40:19.890665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:19.890633    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 19:40:19.891076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:19.890702    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 19:40:20.143218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:20.143116    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:20.552767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:20.552577    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:40:20.552767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:20.552635    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:40:20.618714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:20.600257    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"]
Oct 30 19:40:20.767714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:20.763137    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:21.030713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.028077    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:21.032724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.031855    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:21.035951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.035727    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:40:21.147779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.147746    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"]
Oct 30 19:40:21.164269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.163801    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"]
Oct 30 19:40:21.164269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.163923    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.171035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.171000    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:40:21.171192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.171158    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:21.179779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.178672    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a20b3c92-hr7f6\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.179779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.178751    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a20b3c92-hr7f6\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.179779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.178813    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vbf\" (UniqueName: \"kubernetes.io/projected/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-kube-api-access-f2vbf\") pod \"maas-api-revoke-sub-a20b3c92-hr7f6\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.283410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.283305    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a20b3c92-hr7f6\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.288991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.288957    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a20b3c92-hr7f6\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.289876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.289848    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vbf\" (UniqueName: \"kubernetes.io/projected/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-kube-api-access-f2vbf\") pod \"maas-api-revoke-sub-a20b3c92-hr7f6\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.290712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.285427    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a20b3c92-hr7f6\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.342721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.328426    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a20b3c92-hr7f6\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.375738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.369427    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vbf\" (UniqueName: \"kubernetes.io/projected/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-kube-api-access-f2vbf\") pod \"maas-api-revoke-sub-a20b3c92-hr7f6\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.377755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.376352    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:21.476038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.475988    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:40:21.476241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.476059    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:40:21.489459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.489433    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:21.528857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.528829    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:21.543287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.543223    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:21.824654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.824552    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:21.824888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:21.824814    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"]
Oct 30 19:40:21.829221 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:40:21.829195    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd24c0cb_dd4d_4213_a781_d358c1fbb0b0.slice/crio-ccf6caae81e3514a47e9bce6e6d786d16e4f3211d2b2c8acc995da4a89ffbe8d WatchSource:0}: Error finding container ccf6caae81e3514a47e9bce6e6d786d16e4f3211d2b2c8acc995da4a89ffbe8d: Status 404 returned error can't find the container with id ccf6caae81e3514a47e9bce6e6d786d16e4f3211d2b2c8acc995da4a89ffbe8d
Oct 30 19:40:22.147906 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:40:22.147868    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:40:22.232810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:22.232777    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:22.476191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:22.476114    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:40:22.476191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:22.476166    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:40:22.817767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:22.807044    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6" event={"ID":"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0","Type":"ContainerStarted","Data":"de1ddca24f53b9b463881c13bf7fd0352c164ae418191425ae7f95983ddd990b"}
Oct 30 19:40:22.817767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:22.807103    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6" event={"ID":"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0","Type":"ContainerStarted","Data":"ccf6caae81e3514a47e9bce6e6d786d16e4f3211d2b2c8acc995da4a89ffbe8d"}
Oct 30 19:40:22.851192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:22.851144    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6" podStartSLOduration=1.851130623 podStartE2EDuration="1.851130623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:22.847760291 +0000 UTC m=+2057.750773331" watchObservedRunningTime="2026-10-30 19:40:22.851130623 +0000 UTC m=+2057.754143659"
Oct 30 19:40:23.095675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:23.095458    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:23.671849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:23.671822    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:23.672636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:23.672609    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:23.673826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:23.673806    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:40:23.816007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:23.815977    4925 generic.go:356] "Generic (PLEG): container finished" podID="bd24c0cb-dd4d-4213-a781-d358c1fbb0b0" containerID="de1ddca24f53b9b463881c13bf7fd0352c164ae418191425ae7f95983ddd990b" exitCode=0
Oct 30 19:40:23.816173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:23.816101    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6" event={"ID":"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0","Type":"ContainerDied","Data":"de1ddca24f53b9b463881c13bf7fd0352c164ae418191425ae7f95983ddd990b"}
Oct 30 19:40:24.100772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:24.100560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:24.845217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:24.844885    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:25.089315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.089289    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:25.093971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.093945    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-kube-api-access-f2vbf\" (UniqueName: \"kubernetes.io/projected/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-kube-api-access-f2vbf\") pod \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") "
Oct 30 19:40:25.094217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.094199    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-openshift-ca\") pod \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") "
Oct 30 19:40:25.094364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.094350    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-xks-ca\") pod \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\" (UID: \"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0\") "
Oct 30 19:40:25.094562 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.094541    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-openshift-ca" pod "bd24c0cb-dd4d-4213-a781-d358c1fbb0b0" (UID: "bd24c0cb-dd4d-4213-a781-d358c1fbb0b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:25.095191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.095164    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:25.096502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.096452    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-xks-ca" pod "bd24c0cb-dd4d-4213-a781-d358c1fbb0b0" (UID: "bd24c0cb-dd4d-4213-a781-d358c1fbb0b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:25.096605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.096579    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-kube-api-access-f2vbf" pod "bd24c0cb-dd4d-4213-a781-d358c1fbb0b0" (UID: "bd24c0cb-dd4d-4213-a781-d358c1fbb0b0"). InnerVolumeSpecName "kube-api-access-f2vbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:25.197286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.197253    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:25.197286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.197286    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2vbf\" (UniqueName: \"kubernetes.io/projected/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0-kube-api-access-f2vbf\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:25.520709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.517663    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:25.871075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.870996    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"
Oct 30 19:40:25.871075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.871033    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6" event={"ID":"bd24c0cb-dd4d-4213-a781-d358c1fbb0b0","Type":"ContainerDied","Data":"ccf6caae81e3514a47e9bce6e6d786d16e4f3211d2b2c8acc995da4a89ffbe8d"}
Oct 30 19:40:25.871616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:25.871088    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccf6caae81e3514a47e9bce6e6d786d16e4f3211d2b2c8acc995da4a89ffbe8d"
Oct 30 19:40:26.072852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:26.072817    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"]
Oct 30 19:40:26.293713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:26.291453    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:26.542145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:26.542093    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:27.252109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:27.252077    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:28.075381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:28.075348    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:28.681904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:28.681851    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:29.292941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:29.292256    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:29.954710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:29.951619    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:30.662482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:30.662448    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:31.534716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:31.531256    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:32.709809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:32.709772    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:33.960711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:33.959800    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:34.762823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:34.762777    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:34.797497 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:34.797449    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:34.819529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:34.819486    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"]
Oct 30 19:40:34.831716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:34.825763    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:34.873807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:34.873758    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:34.874623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:34.874589    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:34.909285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:34.909213    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:40:34.931818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:34.931770    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:34.932418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:34.932382    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:40:35.086940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.075982    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.211715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.184805    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.211715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.185178    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Oct 30 19:40:35.274714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.255106    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:35.274714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.258759    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.338214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.338116    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.576717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.550319    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.671175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.553246    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:35.687767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.603191    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.691771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.603233    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.723774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.717801    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.830727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.822838    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.830727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.824156    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:35.830727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.826566    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.832836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.832811    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:40:35.860014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.859959    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" containerID="cri-o://fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674" gracePeriod=45
Oct 30 19:40:35.877799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.877742    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.879574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.879550    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:40:35.929766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.920845    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.929766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.922362    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:40:35.961842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.930223    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.961842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.930473    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:35.961842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:35.947391    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:40:36.028716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.017038    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:40:36.028716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.027316    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:36.028716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.027584    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:40:36.054714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.050652    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:40:36.076893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.076850    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:40:36.116160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.116117    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:40:36.145711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.120965    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:40:36.146345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.135733    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.228382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.188307    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:40:36.228382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.204206    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:40:36.228382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.206321    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Oct 30 19:40:36.228382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.206402    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.259083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.259041    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:40:36.260703 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.260659    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:40:36.341310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.340244    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:36.341786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.341765    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:40:36.426307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.413574    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:40:36.484278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.484138    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:40:36.841770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.753063    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.045751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.951556    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:40:37.046093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.046070    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:40:37.047496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.991801    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:40:37.068073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.991930    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.068734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.992983    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:37.069576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.994134    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:40:37.070220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.997942    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.098811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.999524    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": context deadline exceeded"
Oct 30 19:40:37.099590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:36.991895    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.115705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.069790    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:40:37.115705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.089986    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.120280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.120250    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.124198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.124175    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/health\": context deadline exceeded"
Oct 30 19:40:37.125465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.125447    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:40:37.144840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.144796    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:40:37.233713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.145077    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.292587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.292535    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:37.297711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.293668    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:40:37.310941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.309485    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": dial tcp 10.132.0.190:8000: connect: connection refused"
Oct 30 19:40:37.310941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.309911    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:40:37.519957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.519600    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:40:37.792669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.792592    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:40:37.792669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.792661    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:40:37.793002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.792714    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:40:37.793002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.792744    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:40:37.793002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.792776    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:40:37.793002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.792805    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:40:37.793002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.792835    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:40:37.843484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.843456    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:40:37.882942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.882911    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"]
Oct 30 19:40:37.884324 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.884294    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bd24c0cb-dd4d-4213-a781-d358c1fbb0b0" containerName="revoke-keys"
Oct 30 19:40:37.884541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.884526    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd24c0cb-dd4d-4213-a781-d358c1fbb0b0" containerName="revoke-keys"
Oct 30 19:40:37.884904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.884886    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd24c0cb-dd4d-4213-a781-d358c1fbb0b0" containerName="revoke-keys"
Oct 30 19:40:37.889675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.889648    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:37.899107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.898771    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"]
Oct 30 19:40:37.901585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.901564    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:40:37.909403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.901872    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:37.958161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.958110    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:37.960027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.959995    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vdqp\" (UniqueName: \"kubernetes.io/projected/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-kube-api-access-9vdqp\") pod \"maas-api-revoke-sub-91366b65-qlrq5\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:37.960182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.960158    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91366b65-qlrq5\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:37.960383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:37.960358    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91366b65-qlrq5\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:38.024148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.024113    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": dial tcp 10.132.0.190:8000: connect: connection refused"
Oct 30 19:40:38.062763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.062723    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vdqp\" (UniqueName: \"kubernetes.io/projected/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-kube-api-access-9vdqp\") pod \"maas-api-revoke-sub-91366b65-qlrq5\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:38.063839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.063815    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91366b65-qlrq5\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:38.064825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.064651    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91366b65-qlrq5\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:38.065159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.065140    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91366b65-qlrq5\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:38.074741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.074645    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91366b65-qlrq5\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:38.079245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.079218    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vdqp\" (UniqueName: \"kubernetes.io/projected/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-kube-api-access-9vdqp\") pod \"maas-api-revoke-sub-91366b65-qlrq5\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:38.234278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.233985    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:38.359348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.359321    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:40:38.480163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.480119    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/152df259-957c-42b1-915c-cc6b35b37e7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/152df259-957c-42b1-915c-cc6b35b37e7b-tls-certs\") pod \"152df259-957c-42b1-915c-cc6b35b37e7b\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") "
Oct 30 19:40:38.480356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.480191    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-tmp-dir\") pod \"152df259-957c-42b1-915c-cc6b35b37e7b\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") "
Oct 30 19:40:38.480356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.480252    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-home\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-home\") pod \"152df259-957c-42b1-915c-cc6b35b37e7b\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") "
Oct 30 19:40:38.480356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.480284    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152df259-957c-42b1-915c-cc6b35b37e7b-kube-api-access-m7w9z\" (UniqueName: \"kubernetes.io/projected/152df259-957c-42b1-915c-cc6b35b37e7b-kube-api-access-m7w9z\") pod \"152df259-957c-42b1-915c-cc6b35b37e7b\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") "
Oct 30 19:40:38.480356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.480328    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-model-cache\") pod \"152df259-957c-42b1-915c-cc6b35b37e7b\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") "
Oct 30 19:40:38.480602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.480366    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-dshm\") pod \"152df259-957c-42b1-915c-cc6b35b37e7b\" (UID: \"152df259-957c-42b1-915c-cc6b35b37e7b\") "
Oct 30 19:40:38.480664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.480646    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-home" pod "152df259-957c-42b1-915c-cc6b35b37e7b" (UID: "152df259-957c-42b1-915c-cc6b35b37e7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:38.481496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.481465    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-model-cache" pod "152df259-957c-42b1-915c-cc6b35b37e7b" (UID: "152df259-957c-42b1-915c-cc6b35b37e7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:38.481674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.481644    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-tmp-dir" pod "152df259-957c-42b1-915c-cc6b35b37e7b" (UID: "152df259-957c-42b1-915c-cc6b35b37e7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:38.482429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.482374    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:38.482429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.482396    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:38.482429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.482409    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:38.483782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.483668    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-dshm" pod "152df259-957c-42b1-915c-cc6b35b37e7b" (UID: "152df259-957c-42b1-915c-cc6b35b37e7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:38.486136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.486113    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152df259-957c-42b1-915c-cc6b35b37e7b-tls-certs" pod "152df259-957c-42b1-915c-cc6b35b37e7b" (UID: "152df259-957c-42b1-915c-cc6b35b37e7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:38.493858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.493831    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152df259-957c-42b1-915c-cc6b35b37e7b-kube-api-access-m7w9z" pod "152df259-957c-42b1-915c-cc6b35b37e7b" (UID: "152df259-957c-42b1-915c-cc6b35b37e7b"). InnerVolumeSpecName "kube-api-access-m7w9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:38.584153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.584082    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7w9z\" (UniqueName: \"kubernetes.io/projected/152df259-957c-42b1-915c-cc6b35b37e7b-kube-api-access-m7w9z\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:38.584153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.584111    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/152df259-957c-42b1-915c-cc6b35b37e7b-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:38.584153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.584121    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152df259-957c-42b1-915c-cc6b35b37e7b-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:38.602397 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:40:38.602048    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e391c0_1b15_4f0b_b7c1_6db5d48905c9.slice/crio-2ac9a6d04eb87c630d2a85f35c1fb10978ea889749f546d4bb58654fa3cf8165 WatchSource:0}: Error finding container 2ac9a6d04eb87c630d2a85f35c1fb10978ea889749f546d4bb58654fa3cf8165: Status 404 returned error can't find the container with id 2ac9a6d04eb87c630d2a85f35c1fb10978ea889749f546d4bb58654fa3cf8165
Oct 30 19:40:38.602786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.602682    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"]
Oct 30 19:40:38.766711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.762473    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:38.838421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.838338    4925 generic.go:356] "Generic (PLEG): container finished" podID="152df259-957c-42b1-915c-cc6b35b37e7b" containerID="fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674" exitCode=0
Oct 30 19:40:38.838421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.838388    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"
Oct 30 19:40:38.838639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.838428    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" event={"ID":"152df259-957c-42b1-915c-cc6b35b37e7b","Type":"ContainerDied","Data":"fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674"}
Oct 30 19:40:38.838639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.838470    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf" event={"ID":"152df259-957c-42b1-915c-cc6b35b37e7b","Type":"ContainerDied","Data":"30d7a0e6ca562b8ded1d3e132faacf4eee3515ba5900c8abccfceff2dffdb442"}
Oct 30 19:40:38.838639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.838492    4925 scope.go:122] "RemoveContainer" containerID="fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674"
Oct 30 19:40:38.844092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.840885    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5" event={"ID":"61e391c0-1b15-4f0b-b7c1-6db5d48905c9","Type":"ContainerStarted","Data":"2ac9a6d04eb87c630d2a85f35c1fb10978ea889749f546d4bb58654fa3cf8165"}
Oct 30 19:40:38.864725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.864703    4925 scope.go:122] "RemoveContainer" containerID="fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674"
Oct 30 19:40:38.865069 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:40:38.865046    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674\": container with ID starting with fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674 not found: ID does not exist" containerID="fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674"
Oct 30 19:40:38.865132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.865080    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674"} err="failed to get container status \"fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674\": rpc error: code = NotFound desc = could not find container \"fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674\": container with ID starting with fa68cf29e65f825ebf4a9de34e9ade0a0cc897c0caa7264ee08971e30589d674 not found: ID does not exist"
Oct 30 19:40:38.874514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.874489    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"]
Oct 30 19:40:38.879139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:38.879115    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ddb6ea/auth-test-model-ddb6ea-kserve-6774cf7d8f-252zf"]
Oct 30 19:40:39.145906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:39.145856    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:39.378714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:39.363662    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" path="/var/lib/kubelet/pods/152df259-957c-42b1-915c-cc6b35b37e7b/volumes"
Oct 30 19:40:39.537462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:39.537368    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:39.832474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:39.832391    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk" podUID="1dbbd787-d2fd-4661-ae4d-8e2af9f83f70" containerName="payload-pre-processing" containerID="cri-o://9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6" gracePeriod=25
Oct 30 19:40:39.856371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:39.856344    4925 generic.go:356] "Generic (PLEG): container finished" podID="61e391c0-1b15-4f0b-b7c1-6db5d48905c9" containerID="28149fed15278f4aa1730ef0f08c72f73932f94bebdc5223c12d70ad4bdddd5b" exitCode=0
Oct 30 19:40:39.856511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:39.856427    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5" event={"ID":"61e391c0-1b15-4f0b-b7c1-6db5d48905c9","Type":"ContainerDied","Data":"28149fed15278f4aa1730ef0f08c72f73932f94bebdc5223c12d70ad4bdddd5b"}
Oct 30 19:40:40.159287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.159252    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:40:40.290124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.290094    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-plugins-config-volume\") pod \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\" (UID: \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\") "
Oct 30 19:40:40.290280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.290217    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-kube-api-access-cgvvs\" (UniqueName: \"kubernetes.io/projected/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-kube-api-access-cgvvs\") pod \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\" (UID: \"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70\") "
Oct 30 19:40:40.290423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.290402    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-plugins-config-volume" pod "1dbbd787-d2fd-4661-ae4d-8e2af9f83f70" (UID: "1dbbd787-d2fd-4661-ae4d-8e2af9f83f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:40.290901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.290880    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:40.292522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.292492    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-kube-api-access-cgvvs" pod "1dbbd787-d2fd-4661-ae4d-8e2af9f83f70" (UID: "1dbbd787-d2fd-4661-ae4d-8e2af9f83f70"). InnerVolumeSpecName "kube-api-access-cgvvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:40.393102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.393068    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgvvs\" (UniqueName: \"kubernetes.io/projected/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70-kube-api-access-cgvvs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:40.517763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.517723    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:40.867751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.867655    4925 generic.go:356] "Generic (PLEG): container finished" podID="1dbbd787-d2fd-4661-ae4d-8e2af9f83f70" containerID="9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6" exitCode=0
Oct 30 19:40:40.867751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.867733    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk" event={"ID":"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70","Type":"ContainerDied","Data":"9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6"}
Oct 30 19:40:40.867751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.867747    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"
Oct 30 19:40:40.867988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.867790    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk" event={"ID":"1dbbd787-d2fd-4661-ae4d-8e2af9f83f70","Type":"ContainerDied","Data":"3e59c9584e05c41ab51529b1acb0c94519f4cc7bb076a43c24be0cdf10186a40"}
Oct 30 19:40:40.867988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.867812    4925 scope.go:122] "RemoveContainer" containerID="9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6"
Oct 30 19:40:40.888440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.888413    4925 scope.go:122] "RemoveContainer" containerID="9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6"
Oct 30 19:40:40.890589 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:40:40.890558    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6\": container with ID starting with 9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6 not found: ID does not exist" containerID="9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6"
Oct 30 19:40:40.890720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.890601    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6"} err="failed to get container status \"9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6\": rpc error: code = NotFound desc = could not find container \"9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6\": container with ID starting with 9a4661d6cd4cb438fa0dced9fd5b564af4c64b4dc5d984b8a2bae7076fe49ac6 not found: ID does not exist"
Oct 30 19:40:40.898613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.898586    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"]
Oct 30 19:40:40.902895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:40.902862    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e6199-5fccfbbd79-chtgk"]
Oct 30 19:40:41.051188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.051163    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:41.098644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.098606    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main" containerID="cri-o://05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c" gracePeriod=45
Oct 30 19:40:41.115475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.115449    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-xks-ca\") pod \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") "
Oct 30 19:40:41.115624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.115561    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-kube-api-access-9vdqp\" (UniqueName: \"kubernetes.io/projected/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-kube-api-access-9vdqp\") pod \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") "
Oct 30 19:40:41.115752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.115733    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-openshift-ca\") pod \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\" (UID: \"61e391c0-1b15-4f0b-b7c1-6db5d48905c9\") "
Oct 30 19:40:41.116451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.116418    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-openshift-ca" pod "61e391c0-1b15-4f0b-b7c1-6db5d48905c9" (UID: "61e391c0-1b15-4f0b-b7c1-6db5d48905c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:41.116783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.116755    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:41.117993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.117937    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-kube-api-access-9vdqp" pod "61e391c0-1b15-4f0b-b7c1-6db5d48905c9" (UID: "61e391c0-1b15-4f0b-b7c1-6db5d48905c9"). InnerVolumeSpecName "kube-api-access-9vdqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:41.118807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.118782    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-xks-ca" pod "61e391c0-1b15-4f0b-b7c1-6db5d48905c9" (UID: "61e391c0-1b15-4f0b-b7c1-6db5d48905c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:41.218517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.218493    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:41.218517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.218518    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vdqp\" (UniqueName: \"kubernetes.io/projected/61e391c0-1b15-4f0b-b7c1-6db5d48905c9-kube-api-access-9vdqp\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:41.274477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.274432    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbbd787-d2fd-4661-ae4d-8e2af9f83f70" path="/var/lib/kubelet/pods/1dbbd787-d2fd-4661-ae4d-8e2af9f83f70/volumes"
Oct 30 19:40:41.413728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.411940    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:41.487049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.487021    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:40:41.625331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.625301    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-home\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-home\") pod \"e106ffdc-325c-419f-ae77-5d51a742c185\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") "
Oct 30 19:40:41.625514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.625387    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e106ffdc-325c-419f-ae77-5d51a742c185-tls-certs\" (UniqueName: \"kubernetes.io/secret/e106ffdc-325c-419f-ae77-5d51a742c185-tls-certs\") pod \"e106ffdc-325c-419f-ae77-5d51a742c185\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") "
Oct 30 19:40:41.625514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.625498    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-tmp-dir\") pod \"e106ffdc-325c-419f-ae77-5d51a742c185\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") "
Oct 30 19:40:41.625648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.625547    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-home" pod "e106ffdc-325c-419f-ae77-5d51a742c185" (UID: "e106ffdc-325c-419f-ae77-5d51a742c185"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:41.625648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.625564    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-model-cache\") pod \"e106ffdc-325c-419f-ae77-5d51a742c185\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") "
Oct 30 19:40:41.625648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.625640    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e106ffdc-325c-419f-ae77-5d51a742c185-kube-api-access-ptt88\" (UniqueName: \"kubernetes.io/projected/e106ffdc-325c-419f-ae77-5d51a742c185-kube-api-access-ptt88\") pod \"e106ffdc-325c-419f-ae77-5d51a742c185\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") "
Oct 30 19:40:41.625845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.625815    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-model-cache" pod "e106ffdc-325c-419f-ae77-5d51a742c185" (UID: "e106ffdc-325c-419f-ae77-5d51a742c185"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:41.625900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.625845    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-tmp-dir" pod "e106ffdc-325c-419f-ae77-5d51a742c185" (UID: "e106ffdc-325c-419f-ae77-5d51a742c185"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:41.625976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.625963    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-dshm\") pod \"e106ffdc-325c-419f-ae77-5d51a742c185\" (UID: \"e106ffdc-325c-419f-ae77-5d51a742c185\") "
Oct 30 19:40:41.626652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.626629    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:41.626652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.626652    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:41.626802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.626665    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:41.628081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.628056    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e106ffdc-325c-419f-ae77-5d51a742c185-tls-certs" pod "e106ffdc-325c-419f-ae77-5d51a742c185" (UID: "e106ffdc-325c-419f-ae77-5d51a742c185"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:41.628277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.628259    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-dshm" pod "e106ffdc-325c-419f-ae77-5d51a742c185" (UID: "e106ffdc-325c-419f-ae77-5d51a742c185"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:41.628277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.628261    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e106ffdc-325c-419f-ae77-5d51a742c185-kube-api-access-ptt88" pod "e106ffdc-325c-419f-ae77-5d51a742c185" (UID: "e106ffdc-325c-419f-ae77-5d51a742c185"). InnerVolumeSpecName "kube-api-access-ptt88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:41.728527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.728444    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptt88\" (UniqueName: \"kubernetes.io/projected/e106ffdc-325c-419f-ae77-5d51a742c185-kube-api-access-ptt88\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:41.728527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.728486    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e106ffdc-325c-419f-ae77-5d51a742c185-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:41.728527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.728501    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e106ffdc-325c-419f-ae77-5d51a742c185-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:41.878243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.878204    4925 generic.go:356] "Generic (PLEG): container finished" podID="e106ffdc-325c-419f-ae77-5d51a742c185" containerID="05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c" exitCode=0
Oct 30 19:40:41.878455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.878255    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"
Oct 30 19:40:41.878455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.878291    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" event={"ID":"e106ffdc-325c-419f-ae77-5d51a742c185","Type":"ContainerDied","Data":"05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c"}
Oct 30 19:40:41.878455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.878339    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5" event={"ID":"e106ffdc-325c-419f-ae77-5d51a742c185","Type":"ContainerDied","Data":"4c608e047e59e16ed05bf72374789eefbe1dd5a2d5e4b064fdd5b22646fdbe66"}
Oct 30 19:40:41.878455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.878362    4925 scope.go:122] "RemoveContainer" containerID="05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c"
Oct 30 19:40:41.880052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.880033    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"
Oct 30 19:40:41.880319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.880071    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5" event={"ID":"61e391c0-1b15-4f0b-b7c1-6db5d48905c9","Type":"ContainerDied","Data":"2ac9a6d04eb87c630d2a85f35c1fb10978ea889749f546d4bb58654fa3cf8165"}
Oct 30 19:40:41.880319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.880124    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ac9a6d04eb87c630d2a85f35c1fb10978ea889749f546d4bb58654fa3cf8165"
Oct 30 19:40:41.890993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.890976    4925 scope.go:122] "RemoveContainer" containerID="05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c"
Oct 30 19:40:41.891248 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:40:41.891229    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c\": container with ID starting with 05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c not found: ID does not exist" containerID="05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c"
Oct 30 19:40:41.891323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.891255    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c"} err="failed to get container status \"05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c\": rpc error: code = NotFound desc = could not find container \"05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c\": container with ID starting with 05e8eb2a9749048bdf60fd10014b8cf0057c1fecb17e812869bcc2c3a6f3b71c not found: ID does not exist"
Oct 30 19:40:41.903990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.903967    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"]
Oct 30 19:40:41.908372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:41.908350    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd7c77/auth-test-model-fd7c77-kserve-8696ff8869-zltp5"]
Oct 30 19:40:42.128892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:42.128799    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:43.105607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:43.105576    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:43.266315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:43.266282    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" path="/var/lib/kubelet/pods/e106ffdc-325c-419f-ae77-5d51a742c185/volumes"
Oct 30 19:40:43.891757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:43.891338    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:44.672717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:44.670458    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:45.734420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.734383    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"]
Oct 30 19:40:45.735521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735497    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main"
Oct 30 19:40:45.735576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735526    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main"
Oct 30 19:40:45.735611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735584    4925 cpu_manager.go:414] "containerMap: removing container" podUID="61e391c0-1b15-4f0b-b7c1-6db5d48905c9" containerName="revoke-keys"
Oct 30 19:40:45.735611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735593    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e391c0-1b15-4f0b-b7c1-6db5d48905c9" containerName="revoke-keys"
Oct 30 19:40:45.735674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735621    4925 cpu_manager.go:414] "containerMap: removing container" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main"
Oct 30 19:40:45.735674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735628    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main"
Oct 30 19:40:45.735674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735653    4925 cpu_manager.go:414] "containerMap: removing container" podUID="1dbbd787-d2fd-4661-ae4d-8e2af9f83f70" containerName="payload-pre-processing"
Oct 30 19:40:45.735674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735660    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbbd787-d2fd-4661-ae4d-8e2af9f83f70" containerName="payload-pre-processing"
Oct 30 19:40:45.735903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735886    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e106ffdc-325c-419f-ae77-5d51a742c185" containerName="main"
Oct 30 19:40:45.735938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735917    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e391c0-1b15-4f0b-b7c1-6db5d48905c9" containerName="revoke-keys"
Oct 30 19:40:45.735938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735929    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="152df259-957c-42b1-915c-cc6b35b37e7b" containerName="main"
Oct 30 19:40:45.736001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.735942    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbbd787-d2fd-4661-ae4d-8e2af9f83f70" containerName="payload-pre-processing"
Oct 30 19:40:45.742044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.742018    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:45.753217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.750874    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"]
Oct 30 19:40:45.753366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.753315    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-8f3ff4\"/\"kube-root-ca.crt\""
Oct 30 19:40:45.759958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.759933    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-8f3ff4\"/\"openshift-service-ca.crt\""
Oct 30 19:40:45.764829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.764718    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-8f3ff4\"/\"default-dockercfg-nt95n\""
Oct 30 19:40:45.764829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.764798    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-8f3ff4\"/\"auth-test-model-8f3ff4-kserve-self-signed-certs\""
Oct 30 19:40:45.910921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.910878    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4hlt\" (UniqueName: \"kubernetes.io/projected/36c0c985-99bd-44b0-8869-87286a11ebb1-kube-api-access-l4hlt\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:45.911097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.911040    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-tmp-dir\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:45.911246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.911226    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-dshm\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:45.911316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.911287    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-home\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:45.911381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.911360    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c0c985-99bd-44b0-8869-87286a11ebb1-tls-certs\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:45.911539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:45.911506    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-model-cache\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.019760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.015245    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4hlt\" (UniqueName: \"kubernetes.io/projected/36c0c985-99bd-44b0-8869-87286a11ebb1-kube-api-access-l4hlt\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.019760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.015337    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-tmp-dir\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.019760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.015494    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-dshm\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.019760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.015546    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-home\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.019760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.015588    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c0c985-99bd-44b0-8869-87286a11ebb1-tls-certs\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.019760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.016271    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-model-cache\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.019760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.016889    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-model-cache\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.031715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.021008    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-home\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.031715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.021280    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-tmp-dir\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.031715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.026036    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:46.041732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.035545    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-dshm\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.041732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.038082    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c0c985-99bd-44b0-8869-87286a11ebb1-tls-certs\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.041732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.038763    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4hlt\" (UniqueName: \"kubernetes.io/projected/36c0c985-99bd-44b0-8869-87286a11ebb1-kube-api-access-l4hlt\") pod \"auth-test-model-8f3ff4-kserve-645f64f849-dt7hh\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") " pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.078739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.078190    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:46.508096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.508068    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"]
Oct 30 19:40:46.509528 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:40:46.509503    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36c0c985_99bd_44b0_8869_87286a11ebb1.slice/crio-4946b3333546e9896741ae4e4fecff8a7902ed0693f3a4c078527bcbce9a7ef6 WatchSource:0}: Error finding container 4946b3333546e9896741ae4e4fecff8a7902ed0693f3a4c078527bcbce9a7ef6: Status 404 returned error can't find the container with id 4946b3333546e9896741ae4e4fecff8a7902ed0693f3a4c078527bcbce9a7ef6
Oct 30 19:40:46.782416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.781829    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:46.945210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.944018    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" event={"ID":"36c0c985-99bd-44b0-8869-87286a11ebb1","Type":"ContainerStarted","Data":"d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b"}
Oct 30 19:40:46.945210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.944072    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" event={"ID":"36c0c985-99bd-44b0-8869-87286a11ebb1","Type":"ContainerStarted","Data":"4946b3333546e9896741ae4e4fecff8a7902ed0693f3a4c078527bcbce9a7ef6"}
Oct 30 19:40:46.989976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:46.989928    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" podStartSLOduration=1.9899141519999999 podStartE2EDuration="1.989914152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:46.986895474 +0000 UTC m=+2081.889908513" watchObservedRunningTime="2026-10-30 19:40:46.989914152 +0000 UTC m=+2081.892927189"
Oct 30 19:40:47.079666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:47.079604    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:47.678710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:47.670466    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:48.139581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.139506    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"]
Oct 30 19:40:48.144100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.144081    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.150772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.150750    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:48.150907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.150871    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:40:48.151385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.151363    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"]
Oct 30 19:40:48.188410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.188373    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hkgj\" (UniqueName: \"kubernetes.io/projected/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-kube-api-access-5hkgj\") pod \"maas-api-revoke-sub-83b6a735-7hkkp\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.188594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.188436    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83b6a735-7hkkp\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.188759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.188738    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83b6a735-7hkkp\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.289811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.289781    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hkgj\" (UniqueName: \"kubernetes.io/projected/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-kube-api-access-5hkgj\") pod \"maas-api-revoke-sub-83b6a735-7hkkp\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.289983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.289912    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83b6a735-7hkkp\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.290088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.290073    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83b6a735-7hkkp\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.290725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.290616    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83b6a735-7hkkp\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.292679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.292658    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83b6a735-7hkkp\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.301725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.301670    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hkgj\" (UniqueName: \"kubernetes.io/projected/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-kube-api-access-5hkgj\") pod \"maas-api-revoke-sub-83b6a735-7hkkp\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.462208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.462179    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:48.682071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.680472    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:48.771474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.768981    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"]
Oct 30 19:40:48.961336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:48.961296    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp" event={"ID":"0cd66596-d69c-4f45-9597-f4d6e6d4f86e","Type":"ContainerStarted","Data":"96f069012dd990f01f9232144d2527b68c04d512fd909e5784eaea97c4b0bc48"}
Oct 30 19:40:49.424792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:49.424680    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:49.971920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:49.971879    4925 generic.go:356] "Generic (PLEG): container finished" podID="0cd66596-d69c-4f45-9597-f4d6e6d4f86e" containerID="866a1a5a6adc755823127fa343a9748f0c2c89bae54a1030eb2490ac1e171a7e" exitCode=0
Oct 30 19:40:49.972141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:49.972100    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp" event={"ID":"0cd66596-d69c-4f45-9597-f4d6e6d4f86e","Type":"ContainerDied","Data":"866a1a5a6adc755823127fa343a9748f0c2c89bae54a1030eb2490ac1e171a7e"}
Oct 30 19:40:50.080929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:50.080894    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:50.535714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:50.533224    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:51.209375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.209342    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:51.229205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.229171    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:51.372324 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.372229    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:51.387743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.386149    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-kube-api-access-5hkgj\" (UniqueName: \"kubernetes.io/projected/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-kube-api-access-5hkgj\") pod \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") "
Oct 30 19:40:51.387743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.386266    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-openshift-ca\") pod \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") "
Oct 30 19:40:51.387743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.386553    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-xks-ca\") pod \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\" (UID: \"0cd66596-d69c-4f45-9597-f4d6e6d4f86e\") "
Oct 30 19:40:51.387743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.386999    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-openshift-ca" pod "0cd66596-d69c-4f45-9597-f4d6e6d4f86e" (UID: "0cd66596-d69c-4f45-9597-f4d6e6d4f86e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:51.387743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.387358    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:51.389177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.388979    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-kube-api-access-5hkgj" pod "0cd66596-d69c-4f45-9597-f4d6e6d4f86e" (UID: "0cd66596-d69c-4f45-9597-f4d6e6d4f86e"). InnerVolumeSpecName "kube-api-access-5hkgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:51.389549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.389518    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-xks-ca" pod "0cd66596-d69c-4f45-9597-f4d6e6d4f86e" (UID: "0cd66596-d69c-4f45-9597-f4d6e6d4f86e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:51.489219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.489183    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:51.489219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.489218    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hkgj\" (UniqueName: \"kubernetes.io/projected/0cd66596-d69c-4f45-9597-f4d6e6d4f86e-kube-api-access-5hkgj\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:51.630042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.629944    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:51.808726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:51.808250    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:52.090714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:52.067543    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp" event={"ID":"0cd66596-d69c-4f45-9597-f4d6e6d4f86e","Type":"ContainerDied","Data":"96f069012dd990f01f9232144d2527b68c04d512fd909e5784eaea97c4b0bc48"}
Oct 30 19:40:52.090714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:52.067608    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f069012dd990f01f9232144d2527b68c04d512fd909e5784eaea97c4b0bc48"
Oct 30 19:40:52.090714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:52.067678    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"
Oct 30 19:40:52.219473 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:40:52.219436    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:40:52.340989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:52.339386    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:52.800901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:52.800868    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:53.001712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:52.999156    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:53.569715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:53.568309    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:54.184708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:54.180883    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:55.057294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:55.057254    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:55.914949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:55.911342    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:56.079184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:56.079156    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:56.091986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:56.091953    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:56.274370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:56.274296    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:40:56.935597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:56.935564    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:57.556711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:57.549195    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:57.949921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:57.949885    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:58.152761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:58.152727    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:59.143476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:59.143438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:59.884409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:40:59.884373    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:00.362398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:00.362368    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:00.523508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:00.523472    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:00.692333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:00.692297    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:01.288030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:01.286845    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:02.265743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:02.265704    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:02.886709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:02.883577    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:03.715288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:03.713350    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:04.388097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:04.387474    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:05.032287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:05.032242    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:05.868989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:05.868813    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:06.323342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:06.323312    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:41:06.357632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:06.357603    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:41:06.559718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:06.558801    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:07.201288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:07.201226    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:07.386864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:07.386831    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:08.048091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:08.048055    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:08.787730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:08.787650    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:09.237483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:09.237446    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:09.876106 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:09.876069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:10.034333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:10.034303    4925 scope.go:122] "RemoveContainer" containerID="d408320da678d81d86d3777131195704d417791ea8dffecf1b26b265abb9e56b"
Oct 30 19:41:10.045631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:10.045609    4925 scope.go:122] "RemoveContainer" containerID="64010f5a9fa183c83333409dd587136f7737d21b415e1d81a86faf72a54f3d05"
Oct 30 19:41:10.055490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:10.055460    4925 scope.go:122] "RemoveContainer" containerID="0dc5ed479914afed937d00d1cf0c9e2400ee04d1ae3a769982eae70c95b592ec"
Oct 30 19:41:10.067779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:10.067760    4925 scope.go:122] "RemoveContainer" containerID="d2488e1dff079bf21cc412c36630e73ddcc75698c932c7044de9c922e22b9a2a"
Oct 30 19:41:10.865717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:10.862639    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:11.581384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:11.581346    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:12.213013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:12.209374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:12.948725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:12.948010    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:13.624711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:13.621273    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:14.246769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:14.245718    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:15.023721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:15.023353    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:15.451717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:15.450603    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"]
Oct 30 19:41:15.581710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:15.581318    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:16.019764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:16.017578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:16.222075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:16.222043    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:16.675452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:16.675379    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:16.882722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:16.880238    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:17.209124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:17.209089    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:17.756193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:17.756160    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:19.072316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:19.072278    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:19.807922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:19.807887    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:19.993233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:19.993145    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:20.708862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:20.708812    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded"
Oct 30 19:41:20.765179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:20.765090    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:20.765654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:20.765613    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:41:20.808117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:20.808065    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:20.902947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:20.902891    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:20.978989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:20.978858    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:41:21.078716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.076028    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.118253 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.118205    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:41:21.141723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.119744    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.290772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.286720    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.217:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.336749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.336711    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:21.363866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.363802    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.468924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.467835    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.497163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.496940    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.516833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.509848    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.572156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.572056    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.615074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.615038    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:21.793714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.785951    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:41:21.819080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.796484    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.819740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.818229    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.891066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.891041    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:41:21.891379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.891361    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:41:21.891549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.891537    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:41:21.962954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.962911    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:21.965399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.965373    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:41:22.024217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:21.979805    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.024217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.022671    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:41:22.077121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.077063    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.077890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.077844    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:41:22.116766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.114982    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.116766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.115923    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:41:22.357145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.357053    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"]
Oct 30 19:41:22.402379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.402333    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy" containerID="cri-o://dbcb39cdac533e0e8a5cb8a1635f81d612b9f26009a2f797bc34a825b7c3d738" gracePeriod=30
Oct 30 19:41:22.402952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.394285    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.567735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.401861    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:41:22.593715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.453860    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:41:22.621765 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:22.619675    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:41:22.649712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.628433    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:41:22.666390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.666349    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:41:22.864718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.837808    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.838413    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.856741    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.862984    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.886916    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.888918    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.891674    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.891745    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.891795    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.891829    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.891861    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.891894    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:41:22.903905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.891924    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:41:22.913045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.913007    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:22.967601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:22.967553    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:23.187491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.187416    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:41:23.270765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.270727    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Oct 30 19:41:23.361724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.343518    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:23.439765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.439261    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:23.439765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.439496    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:23.454876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.454838    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.217:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:23.477832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.477789    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:23.570581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.570527    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:23.581726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.571794    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded"
Oct 30 19:41:23.581726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.575271    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:23.595117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.589106    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:41:23.595117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.589167    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:41:23.595117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.589202    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:41:23.652712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.650783    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:41:23.652712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.651890    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:41:23.676281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.672847    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:23.692768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.686902    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:41:23.692768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.691005    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:41:23.726029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.693793    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:41:23.925736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:23.904877    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:41:24.128218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:24.125192    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:41:24.540715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:24.532406    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:25.389062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:25.389022    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:26.194134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:26.194099    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:26.782625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:26.782591    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:27.667436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:27.667397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:28.344362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.344247    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:28.566759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.566724    4925 generic.go:356] "Generic (PLEG): container finished" podID="d234af34-05fd-4440-9419-2a32c3bcef37" containerID="dbcb39cdac533e0e8a5cb8a1635f81d612b9f26009a2f797bc34a825b7c3d738" exitCode=0
Oct 30 19:41:28.567026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.567006    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" event={"ID":"d234af34-05fd-4440-9419-2a32c3bcef37","Type":"ContainerDied","Data":"dbcb39cdac533e0e8a5cb8a1635f81d612b9f26009a2f797bc34a825b7c3d738"}
Oct 30 19:41:28.757879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.739042    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:41:28.785641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.784459    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-certs\") pod \"d234af34-05fd-4440-9419-2a32c3bcef37\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") "
Oct 30 19:41:28.785641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.784513    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-istio-token\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-istio-token\") pod \"d234af34-05fd-4440-9419-2a32c3bcef37\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") "
Oct 30 19:41:28.785641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.784544    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-envoy\") pod \"d234af34-05fd-4440-9419-2a32c3bcef37\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") "
Oct 30 19:41:28.785641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.784739    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d234af34-05fd-4440-9419-2a32c3bcef37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d234af34-05fd-4440-9419-2a32c3bcef37-istiod-ca-cert\") pod \"d234af34-05fd-4440-9419-2a32c3bcef37\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") "
Oct 30 19:41:28.785641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.784773    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-credential-socket\") pod \"d234af34-05fd-4440-9419-2a32c3bcef37\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") "
Oct 30 19:41:28.785641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.784866    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-kube-api-access-ttspj\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-kube-api-access-ttspj\") pod \"d234af34-05fd-4440-9419-2a32c3bcef37\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") "
Oct 30 19:41:28.785641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.784926    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-data\") pod \"d234af34-05fd-4440-9419-2a32c3bcef37\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") "
Oct 30 19:41:28.785641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.784997    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-socket\") pod \"d234af34-05fd-4440-9419-2a32c3bcef37\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") "
Oct 30 19:41:28.785641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.785022    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d234af34-05fd-4440-9419-2a32c3bcef37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d234af34-05fd-4440-9419-2a32c3bcef37-istio-podinfo\") pod \"d234af34-05fd-4440-9419-2a32c3bcef37\" (UID: \"d234af34-05fd-4440-9419-2a32c3bcef37\") "
Oct 30 19:41:28.787387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.787127    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d234af34-05fd-4440-9419-2a32c3bcef37-istiod-ca-cert" pod "d234af34-05fd-4440-9419-2a32c3bcef37" (UID: "d234af34-05fd-4440-9419-2a32c3bcef37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:28.788066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.787586    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-certs" pod "d234af34-05fd-4440-9419-2a32c3bcef37" (UID: "d234af34-05fd-4440-9419-2a32c3bcef37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:28.791959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.788996    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-socket" pod "d234af34-05fd-4440-9419-2a32c3bcef37" (UID: "d234af34-05fd-4440-9419-2a32c3bcef37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:28.791959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.789243    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-data" pod "d234af34-05fd-4440-9419-2a32c3bcef37" (UID: "d234af34-05fd-4440-9419-2a32c3bcef37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:28.791959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.789327    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d234af34-05fd-4440-9419-2a32c3bcef37-istio-podinfo" pod "d234af34-05fd-4440-9419-2a32c3bcef37" (UID: "d234af34-05fd-4440-9419-2a32c3bcef37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:41:28.791959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.789530    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-credential-socket" pod "d234af34-05fd-4440-9419-2a32c3bcef37" (UID: "d234af34-05fd-4440-9419-2a32c3bcef37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:28.806538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.802550    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-istio-token" pod "d234af34-05fd-4440-9419-2a32c3bcef37" (UID: "d234af34-05fd-4440-9419-2a32c3bcef37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:28.806538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.802928    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-envoy" pod "d234af34-05fd-4440-9419-2a32c3bcef37" (UID: "d234af34-05fd-4440-9419-2a32c3bcef37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:28.806538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.805150    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-kube-api-access-ttspj" pod "d234af34-05fd-4440-9419-2a32c3bcef37" (UID: "d234af34-05fd-4440-9419-2a32c3bcef37"). InnerVolumeSpecName "kube-api-access-ttspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:28.813906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.812043    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"]
Oct 30 19:41:28.813906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.813334    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy"
Oct 30 19:41:28.813906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.813363    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy"
Oct 30 19:41:28.813906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.813413    4925 cpu_manager.go:414] "containerMap: removing container" podUID="0cd66596-d69c-4f45-9597-f4d6e6d4f86e" containerName="revoke-keys"
Oct 30 19:41:28.813906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.813421    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd66596-d69c-4f45-9597-f4d6e6d4f86e" containerName="revoke-keys"
Oct 30 19:41:28.813906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.813631    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" containerName="istio-proxy"
Oct 30 19:41:28.813906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.813648    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd66596-d69c-4f45-9597-f4d6e6d4f86e" containerName="revoke-keys"
Oct 30 19:41:28.820652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.820617    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"]
Oct 30 19:41:28.820843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.820805    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:28.825770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.824315    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e6199\""
Oct 30 19:41:28.825770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.825246    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7e6199-dockercfg-9kpn4\""
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888269    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fgf\" (UniqueName: \"kubernetes.io/projected/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-kube-api-access-w8fgf\") pod \"payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm\" (UID: \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888374    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm\" (UID: \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888539    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888550    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888559    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888568    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d234af34-05fd-4440-9419-2a32c3bcef37-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888577    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888585    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttspj\" (UniqueName: \"kubernetes.io/projected/d234af34-05fd-4440-9419-2a32c3bcef37-kube-api-access-ttspj\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888594    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888602    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d234af34-05fd-4440-9419-2a32c3bcef37-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.889523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.888610    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d234af34-05fd-4440-9419-2a32c3bcef37-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:28.990025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.989934    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm\" (UID: \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:28.990255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.990238    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fgf\" (UniqueName: \"kubernetes.io/projected/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-kube-api-access-w8fgf\") pod \"payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm\" (UID: \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:28.990788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:28.990763    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm\" (UID: \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:29.002079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.002055    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fgf\" (UniqueName: \"kubernetes.io/projected/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-kube-api-access-w8fgf\") pod \"payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm\" (UID: \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:29.152968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.152933    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:29.289767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.286024    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:29.583204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.582863    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"
Oct 30 19:41:29.583204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.582947    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv" event={"ID":"d234af34-05fd-4440-9419-2a32c3bcef37","Type":"ContainerDied","Data":"faf81f1a909168daed3874cead20458bc958922ef9840daee2c1b99cbb727e5b"}
Oct 30 19:41:29.583204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.583004    4925 scope.go:122] "RemoveContainer" containerID="dbcb39cdac533e0e8a5cb8a1635f81d612b9f26009a2f797bc34a825b7c3d738"
Oct 30 19:41:29.683435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.683405    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"]
Oct 30 19:41:29.697850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.697818    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"]
Oct 30 19:41:29.698278 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:41:29.698250    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f50cb12_cae3_4cdd_93ca_5bb1c3b7eb34.slice/crio-592e976b0deeb2d05c5333492b58d01725a7189a42f27e96ad22befa5898281d WatchSource:0}: Error finding container 592e976b0deeb2d05c5333492b58d01725a7189a42f27e96ad22befa5898281d: Status 404 returned error can't find the container with id 592e976b0deeb2d05c5333492b58d01725a7189a42f27e96ad22befa5898281d
Oct 30 19:41:29.700772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.700752    4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:41:29.731423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:29.723560    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-9mtsv"]
Oct 30 19:41:30.174402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.174365    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"]
Oct 30 19:41:30.215237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.215208    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.216158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.215101    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"]
Oct 30 19:41:30.262500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.262466    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:30.275927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.275896    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:41:30.307578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.304555    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:30.406747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.406514    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9bcx\" (UniqueName: \"kubernetes.io/projected/533f5474-398a-44c8-b770-15aaf7adb8b5-kube-api-access-w9bcx\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.406747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.406674    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.407054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.406837    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.480415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.480321    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main" containerID="cri-o://c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418" gracePeriod=45
Oct 30 19:41:30.509153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.509078    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9bcx\" (UniqueName: \"kubernetes.io/projected/533f5474-398a-44c8-b770-15aaf7adb8b5-kube-api-access-w9bcx\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.509330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.509233    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.509391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.509327    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.511128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.511097    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.513887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.513857    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.522766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.522736    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9bcx\" (UniqueName: \"kubernetes.io/projected/533f5474-398a-44c8-b770-15aaf7adb8b5-kube-api-access-w9bcx\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.599952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.599910    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm" event={"ID":"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34","Type":"ContainerStarted","Data":"592e976b0deeb2d05c5333492b58d01725a7189a42f27e96ad22befa5898281d"}
Oct 30 19:41:30.635422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.635391    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:30.866232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.865975    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:41:30.959164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.959131    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed24353-f764-42b6-b674-628c07930cca-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed24353-f764-42b6-b674-628c07930cca-tls-certs\") pod \"5ed24353-f764-42b6-b674-628c07930cca\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") "
Oct 30 19:41:30.959464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.959183    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-home\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-home\") pod \"5ed24353-f764-42b6-b674-628c07930cca\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") "
Oct 30 19:41:30.959464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.959254    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-tmp-dir\") pod \"5ed24353-f764-42b6-b674-628c07930cca\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") "
Oct 30 19:41:30.959464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.959294    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-dshm\") pod \"5ed24353-f764-42b6-b674-628c07930cca\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") "
Oct 30 19:41:30.959464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.959383    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed24353-f764-42b6-b674-628c07930cca-kube-api-access-lz7cx\" (UniqueName: \"kubernetes.io/projected/5ed24353-f764-42b6-b674-628c07930cca-kube-api-access-lz7cx\") pod \"5ed24353-f764-42b6-b674-628c07930cca\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") "
Oct 30 19:41:30.959761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.959480    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-home" pod "5ed24353-f764-42b6-b674-628c07930cca" (UID: "5ed24353-f764-42b6-b674-628c07930cca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:30.959761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.959630    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-model-cache\") pod \"5ed24353-f764-42b6-b674-628c07930cca\" (UID: \"5ed24353-f764-42b6-b674-628c07930cca\") "
Oct 30 19:41:30.959949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.959923    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-model-cache" pod "5ed24353-f764-42b6-b674-628c07930cca" (UID: "5ed24353-f764-42b6-b674-628c07930cca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:30.959949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.959935    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-tmp-dir" pod "5ed24353-f764-42b6-b674-628c07930cca" (UID: "5ed24353-f764-42b6-b674-628c07930cca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:30.960621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.960597    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:30.960784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.960627    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:30.960784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.960642    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:30.965235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.965205    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed24353-f764-42b6-b674-628c07930cca-kube-api-access-lz7cx" pod "5ed24353-f764-42b6-b674-628c07930cca" (UID: "5ed24353-f764-42b6-b674-628c07930cca"). InnerVolumeSpecName "kube-api-access-lz7cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:30.965235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.965220    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed24353-f764-42b6-b674-628c07930cca-tls-certs" pod "5ed24353-f764-42b6-b674-628c07930cca" (UID: "5ed24353-f764-42b6-b674-628c07930cca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:30.965448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.965426    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-dshm" pod "5ed24353-f764-42b6-b674-628c07930cca" (UID: "5ed24353-f764-42b6-b674-628c07930cca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:30.970544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:30.970518    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"]
Oct 30 19:41:30.978994 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:41:30.978964    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533f5474_398a_44c8_b770_15aaf7adb8b5.slice/crio-11a93e907a84698ec0699f423835087638882eba660605817b6588348b39dbb3 WatchSource:0}: Error finding container 11a93e907a84698ec0699f423835087638882eba660605817b6588348b39dbb3: Status 404 returned error can't find the container with id 11a93e907a84698ec0699f423835087638882eba660605817b6588348b39dbb3
Oct 30 19:41:31.153826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.153751    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed24353-f764-42b6-b674-628c07930cca-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:31.153826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.153786    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ed24353-f764-42b6-b674-628c07930cca-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:31.153826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.153800    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz7cx\" (UniqueName: \"kubernetes.io/projected/5ed24353-f764-42b6-b674-628c07930cca-kube-api-access-lz7cx\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:31.371914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.370391    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d234af34-05fd-4440-9419-2a32c3bcef37" path="/var/lib/kubelet/pods/d234af34-05fd-4440-9419-2a32c3bcef37/volumes"
Oct 30 19:41:31.611529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.611441    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" event={"ID":"533f5474-398a-44c8-b770-15aaf7adb8b5","Type":"ContainerStarted","Data":"68a382821b572ffcd3e02a5e78983344d8947c651e3adca19add36cb13a8db41"}
Oct 30 19:41:31.611529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.611499    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" event={"ID":"533f5474-398a-44c8-b770-15aaf7adb8b5","Type":"ContainerStarted","Data":"11a93e907a84698ec0699f423835087638882eba660605817b6588348b39dbb3"}
Oct 30 19:41:31.613225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.613199    4925 generic.go:356] "Generic (PLEG): container finished" podID="5ed24353-f764-42b6-b674-628c07930cca" containerID="c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418" exitCode=0
Oct 30 19:41:31.613365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.613244    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"
Oct 30 19:41:31.613365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.613322    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" event={"ID":"5ed24353-f764-42b6-b674-628c07930cca","Type":"ContainerDied","Data":"c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418"}
Oct 30 19:41:31.613365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.613357    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789" event={"ID":"5ed24353-f764-42b6-b674-628c07930cca","Type":"ContainerDied","Data":"8166a5d6e5c146d60a1e2be0b7c0a97b3e7d1dd6b602252ec730d211759131a8"}
Oct 30 19:41:31.613546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.613374    4925 scope.go:122] "RemoveContainer" containerID="c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418"
Oct 30 19:41:31.628501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.628477    4925 scope.go:122] "RemoveContainer" containerID="c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418"
Oct 30 19:41:31.628932 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:31.628898    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418\": container with ID starting with c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418 not found: ID does not exist" containerID="c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418"
Oct 30 19:41:31.629063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.628948    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418"} err="failed to get container status \"c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418\": rpc error: code = NotFound desc = could not find container \"c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418\": container with ID starting with c597af1e3d32d8e3aa894dd4fea67f2cb565449d60a15afdd89815b86147d418 not found: ID does not exist"
Oct 30 19:41:31.631861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.631612    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" podStartSLOduration=1.63159716 podStartE2EDuration="1.63159716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:31.630022586 +0000 UTC m=+2126.533035622" watchObservedRunningTime="2026-10-30 19:41:31.63159716 +0000 UTC m=+2126.534610198"
Oct 30 19:41:31.659751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.659673    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"]
Oct 30 19:41:31.665452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.665420    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-1b1531/auth-test-model-1b1531-kserve-c7698f8c7-vq789"]
Oct 30 19:41:31.852251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:31.852213    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:32.103027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:32.102996    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:32.626059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:32.626019    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm" event={"ID":"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34","Type":"ContainerStarted","Data":"c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386"}
Oct 30 19:41:32.626530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:32.626109    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:32.630170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:32.630146    4925 generic.go:356] "Generic (PLEG): container finished" podID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerID="68a382821b572ffcd3e02a5e78983344d8947c651e3adca19add36cb13a8db41" exitCode=7
Oct 30 19:41:32.630314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:32.630215    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" event={"ID":"533f5474-398a-44c8-b770-15aaf7adb8b5","Type":"ContainerDied","Data":"68a382821b572ffcd3e02a5e78983344d8947c651e3adca19add36cb13a8db41"}
Oct 30 19:41:32.630632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:32.630618    4925 scope.go:122] "RemoveContainer" containerID="68a382821b572ffcd3e02a5e78983344d8947c651e3adca19add36cb13a8db41"
Oct 30 19:41:32.653425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:32.653372    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm" podStartSLOduration=37.949522616 podStartE2EDuration="40.653356169s" totalImagesPullingTime="2.703833553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:32.649020813 +0000 UTC m=+2127.552033881" watchObservedRunningTime="2026-10-30 19:41:32.653356169 +0000 UTC m=+2127.556369208"
Oct 30 19:41:33.097441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.097408    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:33.107717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.107282    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"]
Oct 30 19:41:33.110719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.108449    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main"
Oct 30 19:41:33.110719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.108472    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main"
Oct 30 19:41:33.110719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.108781    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed24353-f764-42b6-b674-628c07930cca" containerName="main"
Oct 30 19:41:33.139742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.139712    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"]
Oct 30 19:41:33.140058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.139904    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:33.169166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.169123    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e50648-dockercfg-7txtz\""
Oct 30 19:41:33.189010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.188959    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e50648\""
Oct 30 19:41:33.241833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.224875    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6h6j\" (UniqueName: \"kubernetes.io/projected/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-kube-api-access-m6h6j\") pod \"payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4\" (UID: \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:33.241833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.225009    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4\" (UID: \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:33.381769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.299049    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:33.381769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.343539    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6h6j\" (UniqueName: \"kubernetes.io/projected/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-kube-api-access-m6h6j\") pod \"payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4\" (UID: \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:33.381769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.343656    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4\" (UID: \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:33.381769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.346217    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4\" (UID: \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:33.564352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.539422    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:33.613699 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.612978    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6h6j\" (UniqueName: \"kubernetes.io/projected/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-kube-api-access-m6h6j\") pod \"payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4\" (UID: \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:33.647233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.647194    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed24353-f764-42b6-b674-628c07930cca" path="/var/lib/kubelet/pods/5ed24353-f764-42b6-b674-628c07930cca/volumes"
Oct 30 19:41:33.781135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:33.781099    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:34.191872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.098524    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"]
Oct 30 19:41:34.191872 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:41:34.099554    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90535d48_03f0_4a14_b3ec_7fc7cd3f9e33.slice/crio-a78616b0e0bae0c186e971bb8280ad1bd29013ececa49324056db163e414d2ba WatchSource:0}: Error finding container a78616b0e0bae0c186e971bb8280ad1bd29013ececa49324056db163e414d2ba: Status 404 returned error can't find the container with id a78616b0e0bae0c186e971bb8280ad1bd29013ececa49324056db163e414d2ba
Oct 30 19:41:34.264921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.263942    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:34.707486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.707448    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4" event={"ID":"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33","Type":"ContainerStarted","Data":"554a07afe33962d3b38c44522524ed78bfdf54a5cd4179d0ea7783a247f6180d"}
Oct 30 19:41:34.708071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.707514    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4" event={"ID":"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33","Type":"ContainerStarted","Data":"a78616b0e0bae0c186e971bb8280ad1bd29013ececa49324056db163e414d2ba"}
Oct 30 19:41:34.708704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.708661    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:34.711548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.711522    4925 generic.go:356] "Generic (PLEG): container finished" podID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerID="bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262" exitCode=0
Oct 30 19:41:34.711761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.711618    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" event={"ID":"7febe178-dee4-4add-9355-5da5c8dfc1ee","Type":"ContainerDied","Data":"bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262"}
Oct 30 19:41:34.712702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.712263    4925 scope.go:122] "RemoveContainer" containerID="bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262"
Oct 30 19:41:34.714065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.714046    4925 generic.go:356] "Generic (PLEG): container finished" podID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerID="618334de20f1a7f7006343909e98c3c4d220763d3cb5eb343eed1879993d7230" exitCode=7
Oct 30 19:41:34.714185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.714095    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" event={"ID":"533f5474-398a-44c8-b770-15aaf7adb8b5","Type":"ContainerDied","Data":"618334de20f1a7f7006343909e98c3c4d220763d3cb5eb343eed1879993d7230"}
Oct 30 19:41:34.714185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.714131    4925 scope.go:122] "RemoveContainer" containerID="68a382821b572ffcd3e02a5e78983344d8947c651e3adca19add36cb13a8db41"
Oct 30 19:41:34.714467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.714448    4925 scope.go:122] "RemoveContainer" containerID="618334de20f1a7f7006343909e98c3c4d220763d3cb5eb343eed1879993d7230"
Oct 30 19:41:34.714733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.714701    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:34.714808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.714790    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:34.714876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.714833    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:34.714876 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:34.714837    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb_odh-ai-gateway-infra(533f5474-398a-44c8-b770-15aaf7adb8b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5"
Oct 30 19:41:34.733269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:34.733230    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4" podStartSLOduration=1.7332176289999999 podStartE2EDuration="1.733217629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:34.727950644 +0000 UTC m=+2129.630963682" watchObservedRunningTime="2026-10-30 19:41:34.733217629 +0000 UTC m=+2129.636230666"
Oct 30 19:41:35.196878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.196582    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:35.731001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.730964    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" event={"ID":"7febe178-dee4-4add-9355-5da5c8dfc1ee","Type":"ContainerStarted","Data":"ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3"}
Oct 30 19:41:35.733193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.733169    4925 scope.go:122] "RemoveContainer" containerID="618334de20f1a7f7006343909e98c3c4d220763d3cb5eb343eed1879993d7230"
Oct 30 19:41:35.733642 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:35.733607    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb_odh-ai-gateway-infra(533f5474-398a-44c8-b770-15aaf7adb8b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5"
Oct 30 19:41:35.734733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.734712    4925 generic.go:356] "Generic (PLEG): container finished" podID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerID="e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd" exitCode=0
Oct 30 19:41:35.734814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.734786    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" event={"ID":"b4afab18-0b83-4db6-a92e-3ad1c5a50da9","Type":"ContainerDied","Data":"e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd"}
Oct 30 19:41:35.735281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.735264    4925 scope.go:122] "RemoveContainer" containerID="e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd"
Oct 30 19:41:35.737076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.737049    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:35.737151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.737116    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:35.737151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.737148    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:35.743081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.743062    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:41:35.788907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.788858    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:41:35.790247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.790214    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:41:35.790368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:35.790273    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Oct 30 19:41:36.095536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:36.092887    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:36.773717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:36.771756    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" event={"ID":"b4afab18-0b83-4db6-a92e-3ad1c5a50da9","Type":"ContainerStarted","Data":"2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47"}
Oct 30 19:41:36.797714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:36.792606    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:41:36.820343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:36.820305    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Oct 30 19:41:36.863135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:36.863099    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:37.290518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:37.290490    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:41:37.291660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:37.291635    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:41:37.291819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:37.291681    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:41:37.422711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:37.422395    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:37.788276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:37.788241    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:41:37.788794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:37.788300    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Oct 30 19:41:38.179729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:38.177230    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:38.291358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:38.291324    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:41:38.291637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:38.291614    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:41:38.741713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:38.740914    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:41:38.788416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:38.788392    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:41:38.788832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:38.788772    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Oct 30 19:41:38.915713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:38.915616    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:39.290257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:39.290224    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:41:39.290457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:39.290272    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:41:39.461750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:39.458884    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:39.788431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:39.788396    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:41:39.788973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:39.788446    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Oct 30 19:41:40.291199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:40.291151    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:41:40.291540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:40.291429    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:41:40.369579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:40.369542    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:40.564964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:40.560131    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:40.812179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:40.812142    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Oct 30 19:41:40.829108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:40.828877    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Oct 30 19:41:41.292287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:41.291836    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:41:41.292287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:41.291917    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:41:41.349728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:41.346354    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:41.793031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:41.793002    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:41:41.794282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:41.794265    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:41:42.186488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:42.186450    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:42.290396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:42.290363    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 19:41:42.290573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:42.290415    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 19:41:43.110710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:43.110236    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:43.530725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:43.522901    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:41:43.530725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:43.525000    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:41:43.531411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:43.530918    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:41:43.684322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:43.684293    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:41:44.231710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:44.231233    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:44.935480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:44.935436    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.053490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:45.053459    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:45.282037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:45.281954    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:45.369847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:45.369804    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.217:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.590552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:45.589758    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:45.886774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:45.886659    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:46.255821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:46.255794    4925 scope.go:122] "RemoveContainer" containerID="618334de20f1a7f7006343909e98c3c4d220763d3cb5eb343eed1879993d7230"
Oct 30 19:41:46.711082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:46.711051    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:46.775443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:46.775360    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:41:46.776138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:46.776114    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8879-f26104"]
Oct 30 19:41:46.779832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:46.779809    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:46.784908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:46.784888    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8879-f26104"]
Oct 30 19:41:46.788189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:46.788155    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bmmlc\""
Oct 30 19:41:46.894584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:46.894552    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbmb\" (UniqueName: \"kubernetes.io/projected/af74df5c-714f-40a7-bb3c-9219fd63a4b3-kube-api-access-vlbmb\") pod \"test-curl-8879-f26104\" (UID: \"af74df5c-714f-40a7-bb3c-9219fd63a4b3\") " pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:46.996805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:46.996768    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbmb\" (UniqueName: \"kubernetes.io/projected/af74df5c-714f-40a7-bb3c-9219fd63a4b3-kube-api-access-vlbmb\") pod \"test-curl-8879-f26104\" (UID: \"af74df5c-714f-40a7-bb3c-9219fd63a4b3\") " pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:47.008540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:47.008514    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" event={"ID":"533f5474-398a-44c8-b770-15aaf7adb8b5","Type":"ContainerStarted","Data":"343133887c4c202f69489c33e684db4262ee9494d9497739666575dd037a3791"}
Oct 30 19:41:47.009025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:47.009006    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbmb\" (UniqueName: \"kubernetes.io/projected/af74df5c-714f-40a7-bb3c-9219fd63a4b3-kube-api-access-vlbmb\") pod \"test-curl-8879-f26104\" (UID: \"af74df5c-714f-40a7-bb3c-9219fd63a4b3\") " pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:47.045354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:47.045294    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"]
Oct 30 19:41:47.049635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:47.049612    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0109f6-4qk9n"]
Oct 30 19:41:47.093863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:47.093833    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:47.266477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:47.266444    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a5e0b3-69d1-4c92-981d-c6e8526a98b4" path="/var/lib/kubelet/pods/30a5e0b3-69d1-4c92-981d-c6e8526a98b4/volumes"
Oct 30 19:41:47.401001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:47.400973    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8879-f26104"]
Oct 30 19:41:47.401357 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:41:47.401330    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf74df5c_714f_40a7_bb3c_9219fd63a4b3.slice/crio-9b22f59671d5b7acb58e235e7dfd859ead08df43f00b6bc74b34f3842e44945b WatchSource:0}: Error finding container 9b22f59671d5b7acb58e235e7dfd859ead08df43f00b6bc74b34f3842e44945b: Status 404 returned error can't find the container with id 9b22f59671d5b7acb58e235e7dfd859ead08df43f00b6bc74b34f3842e44945b
Oct 30 19:41:47.605345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:47.600090    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:48.019245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.019204    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8879-f26104" event={"ID":"af74df5c-714f-40a7-bb3c-9219fd63a4b3","Type":"ContainerStarted","Data":"7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1"}
Oct 30 19:41:48.019445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.019269    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8879-f26104" event={"ID":"af74df5c-714f-40a7-bb3c-9219fd63a4b3","Type":"ContainerStarted","Data":"9b22f59671d5b7acb58e235e7dfd859ead08df43f00b6bc74b34f3842e44945b"}
Oct 30 19:41:48.021387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.021359    4925 generic.go:356] "Generic (PLEG): container finished" podID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerID="343133887c4c202f69489c33e684db4262ee9494d9497739666575dd037a3791" exitCode=7
Oct 30 19:41:48.021495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.021424    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" event={"ID":"533f5474-398a-44c8-b770-15aaf7adb8b5","Type":"ContainerDied","Data":"343133887c4c202f69489c33e684db4262ee9494d9497739666575dd037a3791"}
Oct 30 19:41:48.021557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.021512    4925 scope.go:122] "RemoveContainer" containerID="618334de20f1a7f7006343909e98c3c4d220763d3cb5eb343eed1879993d7230"
Oct 30 19:41:48.021945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.021925    4925 scope.go:122] "RemoveContainer" containerID="343133887c4c202f69489c33e684db4262ee9494d9497739666575dd037a3791"
Oct 30 19:41:48.022339 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:48.022320    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb_odh-ai-gateway-infra(533f5474-398a-44c8-b770-15aaf7adb8b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5"
Oct 30 19:41:48.043006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.042984    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"]
Oct 30 19:41:48.048044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.048005    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8879-f26104" podStartSLOduration=2.047992548 podStartE2EDuration="2.047992548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:48.044966224 +0000 UTC m=+2142.947979260" watchObservedRunningTime="2026-10-30 19:41:48.047992548 +0000 UTC m=+2142.951005581"
Oct 30 19:41:48.200473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.200436    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:48.374916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.374819    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:48.727385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.727304    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:48.906712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:48.906121    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:49.328761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.328726    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:49.410796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.398704    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8879-f26104"]
Oct 30 19:41:49.411278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.411025    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8879-f26104"]
Oct 30 19:41:49.533936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.533909    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:49.553665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.553636    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8879-38227f"]
Oct 30 19:41:49.554757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.554734    4925 cpu_manager.go:414] "containerMap: removing container" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerName="revoke-keys"
Oct 30 19:41:49.554757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.554758    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerName="revoke-keys"
Oct 30 19:41:49.554979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.554774    4925 cpu_manager.go:414] "containerMap: removing container" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerName="revoke-keys"
Oct 30 19:41:49.554979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.554779    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerName="revoke-keys"
Oct 30 19:41:49.554979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.554789    4925 cpu_manager.go:414] "containerMap: removing container" podUID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" containerName="test-curl-8879-f26104"
Oct 30 19:41:49.554979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.554795    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" containerName="test-curl-8879-f26104"
Oct 30 19:41:49.555202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.555067    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerName="revoke-keys"
Oct 30 19:41:49.555202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.555081    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerName="revoke-keys"
Oct 30 19:41:49.555202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.555089    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" containerName="test-curl-8879-f26104"
Oct 30 19:41:49.558002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.557978    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:49.566250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.566207    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8879-38227f"]
Oct 30 19:41:49.644159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.644130    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-openshift-ca\") pod \"533f5474-398a-44c8-b770-15aaf7adb8b5\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") "
Oct 30 19:41:49.644355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.644262    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-xks-ca\") pod \"533f5474-398a-44c8-b770-15aaf7adb8b5\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") "
Oct 30 19:41:49.644355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.644293    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533f5474-398a-44c8-b770-15aaf7adb8b5-kube-api-access-w9bcx\" (UniqueName: \"kubernetes.io/projected/533f5474-398a-44c8-b770-15aaf7adb8b5-kube-api-access-w9bcx\") pod \"533f5474-398a-44c8-b770-15aaf7adb8b5\" (UID: \"533f5474-398a-44c8-b770-15aaf7adb8b5\") "
Oct 30 19:41:49.644551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.644525    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-openshift-ca" pod "533f5474-398a-44c8-b770-15aaf7adb8b5" (UID: "533f5474-398a-44c8-b770-15aaf7adb8b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:49.644838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.644814    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97jj\" (UniqueName: \"kubernetes.io/projected/d9fae881-0cda-482f-8b46-10455c84e7f4-kube-api-access-x97jj\") pod \"test-curl-8879-38227f\" (UID: \"d9fae881-0cda-482f-8b46-10455c84e7f4\") " pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:49.645323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.645302    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:49.646652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.646626    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-xks-ca" pod "533f5474-398a-44c8-b770-15aaf7adb8b5" (UID: "533f5474-398a-44c8-b770-15aaf7adb8b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:49.646779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.646676    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533f5474-398a-44c8-b770-15aaf7adb8b5-kube-api-access-w9bcx" pod "533f5474-398a-44c8-b770-15aaf7adb8b5" (UID: "533f5474-398a-44c8-b770-15aaf7adb8b5"). InnerVolumeSpecName "kube-api-access-w9bcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:49.746752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.746656    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97jj\" (UniqueName: \"kubernetes.io/projected/d9fae881-0cda-482f-8b46-10455c84e7f4-kube-api-access-x97jj\") pod \"test-curl-8879-38227f\" (UID: \"d9fae881-0cda-482f-8b46-10455c84e7f4\") " pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:49.747056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.747035    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533f5474-398a-44c8-b770-15aaf7adb8b5-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:49.747134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.747062    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9bcx\" (UniqueName: \"kubernetes.io/projected/533f5474-398a-44c8-b770-15aaf7adb8b5-kube-api-access-w9bcx\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:49.759082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.759059    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97jj\" (UniqueName: \"kubernetes.io/projected/d9fae881-0cda-482f-8b46-10455c84e7f4-kube-api-access-x97jj\") pod \"test-curl-8879-38227f\" (UID: \"d9fae881-0cda-482f-8b46-10455c84e7f4\") " pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:49.903115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.903078    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:49.979573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:49.977910    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:50.159440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:50.159407    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"
Oct 30 19:41:50.159440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:50.159420    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb" event={"ID":"533f5474-398a-44c8-b770-15aaf7adb8b5","Type":"ContainerDied","Data":"11a93e907a84698ec0699f423835087638882eba660605817b6588348b39dbb3"}
Oct 30 19:41:50.159666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:50.159531    4925 scope.go:122] "RemoveContainer" containerID="343133887c4c202f69489c33e684db4262ee9494d9497739666575dd037a3791"
Oct 30 19:41:50.159752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:50.159706    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8879-f26104" podUID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" containerName="test-curl-8879-f26104" containerID="cri-o://7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1" gracePeriod=2
Oct 30 19:41:50.165040 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:50.165014    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-f26104\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:50.166714 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:50.166666    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-f26104\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:50.175148 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:50.175124    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-f26104\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:50.194164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:50.194141    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"]
Oct 30 19:41:50.201916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:50.201893    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-9sthb"]
Oct 30 19:41:50.248677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:50.248654    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8879-38227f"]
Oct 30 19:41:50.250200 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:41:50.250176    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9fae881_0cda_482f_8b46_10455c84e7f4.slice/crio-c00763e35872bd7cd6ae82b516780842638e611750dd7e7bf00da9ce53768e73 WatchSource:0}: Error finding container c00763e35872bd7cd6ae82b516780842638e611750dd7e7bf00da9ce53768e73: Status 404 returned error can't find the container with id c00763e35872bd7cd6ae82b516780842638e611750dd7e7bf00da9ce53768e73
Oct 30 19:41:50.685403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:50.680502    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:51.169306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:51.169266    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8879-38227f" event={"ID":"d9fae881-0cda-482f-8b46-10455c84e7f4","Type":"ContainerStarted","Data":"5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e"}
Oct 30 19:41:51.169485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:51.169320    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8879-38227f" event={"ID":"d9fae881-0cda-482f-8b46-10455c84e7f4","Type":"ContainerStarted","Data":"c00763e35872bd7cd6ae82b516780842638e611750dd7e7bf00da9ce53768e73"}
Oct 30 19:41:51.173161 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:51.173130    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-f26104\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:51.187921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:51.187879    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8879-38227f" podStartSLOduration=2.187866031 podStartE2EDuration="2.187866031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:51.186325059 +0000 UTC m=+2146.089338097" watchObservedRunningTime="2026-10-30 19:41:51.187866031 +0000 UTC m=+2146.090879068"
Oct 30 19:41:51.265493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:51.265463    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" path="/var/lib/kubelet/pods/533f5474-398a-44c8-b770-15aaf7adb8b5/volumes"
Oct 30 19:41:51.608768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:51.608364    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:51.779958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:51.779931    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:51.979587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:51.979550    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8879-38227f"]
Oct 30 19:41:51.983220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:51.983177    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:51.986643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:51.986614    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8879-38227f"]
Oct 30 19:41:52.143042 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:52.142980    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:41:52.531556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.531493    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:52.622809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.622777    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af74df5c-714f-40a7-bb3c-9219fd63a4b3-kube-api-access-vlbmb\" (UniqueName: \"kubernetes.io/projected/af74df5c-714f-40a7-bb3c-9219fd63a4b3-kube-api-access-vlbmb\") pod \"af74df5c-714f-40a7-bb3c-9219fd63a4b3\" (UID: \"af74df5c-714f-40a7-bb3c-9219fd63a4b3\") "
Oct 30 19:41:52.625259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.625227    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af74df5c-714f-40a7-bb3c-9219fd63a4b3-kube-api-access-vlbmb" pod "af74df5c-714f-40a7-bb3c-9219fd63a4b3" (UID: "af74df5c-714f-40a7-bb3c-9219fd63a4b3"). InnerVolumeSpecName "kube-api-access-vlbmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:52.726711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.725803    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlbmb\" (UniqueName: \"kubernetes.io/projected/af74df5c-714f-40a7-bb3c-9219fd63a4b3-kube-api-access-vlbmb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.768775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.732931    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"]
Oct 30 19:41:52.768775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.741245    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:52.768775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.762197    4925 cpu_manager.go:414] "containerMap: removing container" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerName="revoke-keys"
Oct 30 19:41:52.768775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.762231    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerName="revoke-keys"
Oct 30 19:41:52.768775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.762271    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" containerName="test-curl-8879-38227f"
Oct 30 19:41:52.768775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.762281    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" containerName="test-curl-8879-38227f"
Oct 30 19:41:52.768775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.762484    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="533f5474-398a-44c8-b770-15aaf7adb8b5" containerName="revoke-keys"
Oct 30 19:41:52.768775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.762500    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" containerName="test-curl-8879-38227f"
Oct 30 19:41:52.779071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.772486    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"]
Oct 30 19:41:52.779071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.772618    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:52.784859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.783135    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:52.784859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.783270    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:41:52.827288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.827256    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9a48d5-vdx9q\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:52.827576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.827547    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7khn\" (UniqueName: \"kubernetes.io/projected/428eb372-ec1f-4905-84e6-983bfd0cc5c6-kube-api-access-c7khn\") pod \"maas-api-revoke-sub-0e9a48d5-vdx9q\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:52.827789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.827768    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9a48d5-vdx9q\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:52.929058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.929026    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7khn\" (UniqueName: \"kubernetes.io/projected/428eb372-ec1f-4905-84e6-983bfd0cc5c6-kube-api-access-c7khn\") pod \"maas-api-revoke-sub-0e9a48d5-vdx9q\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:52.929220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.929157    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9a48d5-vdx9q\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:52.929314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.929297    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9a48d5-vdx9q\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:52.930450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.930418    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9a48d5-vdx9q\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:52.935110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.935082    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9a48d5-vdx9q\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:52.945087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:52.945058    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7khn\" (UniqueName: \"kubernetes.io/projected/428eb372-ec1f-4905-84e6-983bfd0cc5c6-kube-api-access-c7khn\") pod \"maas-api-revoke-sub-0e9a48d5-vdx9q\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:53.097952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.097866    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:53.251462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.251416    4925 generic.go:356] "Generic (PLEG): container finished" podID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" containerID="7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1" exitCode=137
Oct 30 19:41:53.251462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.251456    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-f26104"
Oct 30 19:41:53.252195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.251651    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8879-38227f" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" containerName="test-curl-8879-38227f" containerID="cri-o://5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e" gracePeriod=2
Oct 30 19:41:53.252195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.251794    4925 scope.go:122] "RemoveContainer" containerID="7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1"
Oct 30 19:41:53.253987 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:53.253945    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-38227f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:53.260962 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:53.260933    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-38227f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:53.265834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.265815    4925 scope.go:122] "RemoveContainer" containerID="7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1"
Oct 30 19:41:53.266244 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:53.266151    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1\": container with ID starting with 7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1 not found: ID does not exist" containerID="7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1"
Oct 30 19:41:53.266244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.266184    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1"} err="failed to get container status \"7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1\": rpc error: code = NotFound desc = could not find container \"7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1\": container with ID starting with 7c60071e21b007b5e2435e4ec2de655c4001fd14a8e575b5c5699f2106ee10f1 not found: ID does not exist"
Oct 30 19:41:53.270501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.270440    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af74df5c-714f-40a7-bb3c-9219fd63a4b3" path="/var/lib/kubelet/pods/af74df5c-714f-40a7-bb3c-9219fd63a4b3/volumes"
Oct 30 19:41:53.322710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.322242    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:53.393631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:53.393604    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"]
Oct 30 19:41:53.395249 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:41:53.395220    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428eb372_ec1f_4905_84e6_983bfd0cc5c6.slice/crio-fc23b019611cf72e8c90867f3594d131ed87bb5516123bbc27489f07d04fdb4c WatchSource:0}: Error finding container fc23b019611cf72e8c90867f3594d131ed87bb5516123bbc27489f07d04fdb4c: Status 404 returned error can't find the container with id fc23b019611cf72e8c90867f3594d131ed87bb5516123bbc27489f07d04fdb4c
Oct 30 19:41:54.046756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:54.041823    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:54.263169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:54.263136    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q" event={"ID":"428eb372-ec1f-4905-84e6-983bfd0cc5c6","Type":"ContainerStarted","Data":"cd4693cd0f1a133eebd3b58f0f9353dde34cdb14844025a7786c1faa87a8a6c8"}
Oct 30 19:41:54.263346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:54.263194    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q" event={"ID":"428eb372-ec1f-4905-84e6-983bfd0cc5c6","Type":"ContainerStarted","Data":"fc23b019611cf72e8c90867f3594d131ed87bb5516123bbc27489f07d04fdb4c"}
Oct 30 19:41:54.286452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:54.286403    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q" podStartSLOduration=2.286390244 podStartE2EDuration="2.286390244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:54.281147574 +0000 UTC m=+2149.184160611" watchObservedRunningTime="2026-10-30 19:41:54.286390244 +0000 UTC m=+2149.189403281"
Oct 30 19:41:54.770981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:54.770815    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:55.276086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:55.276045    4925 generic.go:356] "Generic (PLEG): container finished" podID="428eb372-ec1f-4905-84e6-983bfd0cc5c6" containerID="cd4693cd0f1a133eebd3b58f0f9353dde34cdb14844025a7786c1faa87a8a6c8" exitCode=0
Oct 30 19:41:55.276409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:55.276135    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q" event={"ID":"428eb372-ec1f-4905-84e6-983bfd0cc5c6","Type":"ContainerDied","Data":"cd4693cd0f1a133eebd3b58f0f9353dde34cdb14844025a7786c1faa87a8a6c8"}
Oct 30 19:41:55.483530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:55.483504    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:55.485754 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:55.485719    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-38227f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:55.585966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:55.585881    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9fae881-0cda-482f-8b46-10455c84e7f4-kube-api-access-x97jj\" (UniqueName: \"kubernetes.io/projected/d9fae881-0cda-482f-8b46-10455c84e7f4-kube-api-access-x97jj\") pod \"d9fae881-0cda-482f-8b46-10455c84e7f4\" (UID: \"d9fae881-0cda-482f-8b46-10455c84e7f4\") "
Oct 30 19:41:55.588183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:55.588159    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9fae881-0cda-482f-8b46-10455c84e7f4-kube-api-access-x97jj" pod "d9fae881-0cda-482f-8b46-10455c84e7f4" (UID: "d9fae881-0cda-482f-8b46-10455c84e7f4"). InnerVolumeSpecName "kube-api-access-x97jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:55.688780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:55.688748    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x97jj\" (UniqueName: \"kubernetes.io/projected/d9fae881-0cda-482f-8b46-10455c84e7f4-kube-api-access-x97jj\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:55.805128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:55.805090    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:56.286748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.286704    4925 generic.go:356] "Generic (PLEG): container finished" podID="d9fae881-0cda-482f-8b46-10455c84e7f4" containerID="5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e" exitCode=137
Oct 30 19:41:56.287235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.286752    4925 scope.go:122] "RemoveContainer" containerID="5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e"
Oct 30 19:41:56.287235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.286776    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:56.288806 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:56.288772    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-38227f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:56.297197 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:56.297175    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-38227f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:56.300254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.300238    4925 scope.go:122] "RemoveContainer" containerID="5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e"
Oct 30 19:41:56.300595 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:56.300565    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e\": container with ID starting with 5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e not found: ID does not exist" containerID="5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e"
Oct 30 19:41:56.300664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.300606    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e"} err="failed to get container status \"5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e\": rpc error: code = NotFound desc = could not find container \"5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e\": container with ID starting with 5aba968f788d7f7620b03e52c3adcab11828a63d1c6fe9c1628d1cf122bd6d3e not found: ID does not exist"
Oct 30 19:41:56.392759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.392652    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:56.537167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.537103    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:56.538993 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:41:56.538968    4925 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8879-38227f\" is forbidden: User \"system:node:ip-10-0-130-164.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-164.us-east-2.compute.internal' and this object" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" pod="opendatahub/test-curl-8879-38227f"
Oct 30 19:41:56.604728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.604681    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-xks-ca\") pod \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") "
Oct 30 19:41:56.604942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.604790    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428eb372-ec1f-4905-84e6-983bfd0cc5c6-kube-api-access-c7khn\" (UniqueName: \"kubernetes.io/projected/428eb372-ec1f-4905-84e6-983bfd0cc5c6-kube-api-access-c7khn\") pod \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") "
Oct 30 19:41:56.604942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.604863    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-openshift-ca\") pod \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\" (UID: \"428eb372-ec1f-4905-84e6-983bfd0cc5c6\") "
Oct 30 19:41:56.605243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.605216    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-openshift-ca" pod "428eb372-ec1f-4905-84e6-983bfd0cc5c6" (UID: "428eb372-ec1f-4905-84e6-983bfd0cc5c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:56.605904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.605879    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:56.607066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.607041    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-xks-ca" pod "428eb372-ec1f-4905-84e6-983bfd0cc5c6" (UID: "428eb372-ec1f-4905-84e6-983bfd0cc5c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:56.607418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.607398    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428eb372-ec1f-4905-84e6-983bfd0cc5c6-kube-api-access-c7khn" pod "428eb372-ec1f-4905-84e6-983bfd0cc5c6" (UID: "428eb372-ec1f-4905-84e6-983bfd0cc5c6"). InnerVolumeSpecName "kube-api-access-c7khn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:56.707611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.707576    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428eb372-ec1f-4905-84e6-983bfd0cc5c6-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:56.707611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:56.707605    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7khn\" (UniqueName: \"kubernetes.io/projected/428eb372-ec1f-4905-84e6-983bfd0cc5c6-kube-api-access-c7khn\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:57.123297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:57.123263    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:57.266085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:57.266053    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9fae881-0cda-482f-8b46-10455c84e7f4" path="/var/lib/kubelet/pods/d9fae881-0cda-482f-8b46-10455c84e7f4/volumes"
Oct 30 19:41:57.297533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:57.297505    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"
Oct 30 19:41:57.297533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:57.297525    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q" event={"ID":"428eb372-ec1f-4905-84e6-983bfd0cc5c6","Type":"ContainerDied","Data":"fc23b019611cf72e8c90867f3594d131ed87bb5516123bbc27489f07d04fdb4c"}
Oct 30 19:41:57.297982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:57.297576    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc23b019611cf72e8c90867f3594d131ed87bb5516123bbc27489f07d04fdb4c"
Oct 30 19:41:57.763712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:57.746620    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:58.073855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:58.073770    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:58.487814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:58.483834    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:59.269713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:59.267814    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:59.560493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:41:59.560402    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:00.045951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:00.040036    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:00.655732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:00.647565    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:01.284506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:01.284461    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:02.120052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:02.120017    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:02.796870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:02.796145    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:03.001837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.001799    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"]
Oct 30 19:42:03.002582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.002567    4925 cpu_manager.go:414] "containerMap: removing container" podUID="428eb372-ec1f-4905-84e6-983bfd0cc5c6" containerName="revoke-keys"
Oct 30 19:42:03.002632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.002584    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428eb372-ec1f-4905-84e6-983bfd0cc5c6" containerName="revoke-keys"
Oct 30 19:42:03.002734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.002724    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="428eb372-ec1f-4905-84e6-983bfd0cc5c6" containerName="revoke-keys"
Oct 30 19:42:03.005426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.005409    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.013736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.012195    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:42:03.013736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.012263    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:03.016513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.015366    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"]
Oct 30 19:42:03.083331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.083244    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affab8b4-2jjhw\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.083546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.083416    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affab8b4-2jjhw\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.083617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.083601    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spg2k\" (UniqueName: \"kubernetes.io/projected/93c79bd8-12e7-4c04-9774-e2c638b2548f-kube-api-access-spg2k\") pod \"maas-api-revoke-sub-affab8b4-2jjhw\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.185650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.185615    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spg2k\" (UniqueName: \"kubernetes.io/projected/93c79bd8-12e7-4c04-9774-e2c638b2548f-kube-api-access-spg2k\") pod \"maas-api-revoke-sub-affab8b4-2jjhw\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.186198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.185762    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affab8b4-2jjhw\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.186198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.185877    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affab8b4-2jjhw\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.186603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.186582    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affab8b4-2jjhw\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.188486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.188463    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affab8b4-2jjhw\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.201140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.201114    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spg2k\" (UniqueName: \"kubernetes.io/projected/93c79bd8-12e7-4c04-9774-e2c638b2548f-kube-api-access-spg2k\") pod \"maas-api-revoke-sub-affab8b4-2jjhw\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.322384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.322351    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:03.507144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.507104    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:03.617012 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:42:03.616981    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c79bd8_12e7_4c04_9774_e2c638b2548f.slice/crio-bc60f3f4f8cff1a475b3b31153e34d2db5dab7734226e138c9318cf4b37e3ac1 WatchSource:0}: Error finding container bc60f3f4f8cff1a475b3b31153e34d2db5dab7734226e138c9318cf4b37e3ac1: Status 404 returned error can't find the container with id bc60f3f4f8cff1a475b3b31153e34d2db5dab7734226e138c9318cf4b37e3ac1
Oct 30 19:42:03.617383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:03.617359    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"]
Oct 30 19:42:04.172710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:04.168616    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:04.362122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:04.362034    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw" event={"ID":"93c79bd8-12e7-4c04-9774-e2c638b2548f","Type":"ContainerStarted","Data":"918f29f6a25b789ffef48fa0b3076d96b5780c1245bc22f5837e3addb9036014"}
Oct 30 19:42:04.362122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:04.362091    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw" event={"ID":"93c79bd8-12e7-4c04-9774-e2c638b2548f","Type":"ContainerStarted","Data":"bc60f3f4f8cff1a475b3b31153e34d2db5dab7734226e138c9318cf4b37e3ac1"}
Oct 30 19:42:04.379682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:04.379629    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw" podStartSLOduration=2.379614739 podStartE2EDuration="2.379614739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:04.377905809 +0000 UTC m=+2159.280918846" watchObservedRunningTime="2026-10-30 19:42:04.379614739 +0000 UTC m=+2159.282627810"
Oct 30 19:42:04.811920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:04.811316    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:05.372140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:05.372108    4925 generic.go:356] "Generic (PLEG): container finished" podID="93c79bd8-12e7-4c04-9774-e2c638b2548f" containerID="918f29f6a25b789ffef48fa0b3076d96b5780c1245bc22f5837e3addb9036014" exitCode=0
Oct 30 19:42:05.372538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:05.372160    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw" event={"ID":"93c79bd8-12e7-4c04-9774-e2c638b2548f","Type":"ContainerDied","Data":"918f29f6a25b789ffef48fa0b3076d96b5780c1245bc22f5837e3addb9036014"}
Oct 30 19:42:05.507710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:05.502664    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:06.157833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.156444    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:06.560294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.560265    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:06.684543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.684512    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c79bd8-12e7-4c04-9774-e2c638b2548f-kube-api-access-spg2k\" (UniqueName: \"kubernetes.io/projected/93c79bd8-12e7-4c04-9774-e2c638b2548f-kube-api-access-spg2k\") pod \"93c79bd8-12e7-4c04-9774-e2c638b2548f\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") "
Oct 30 19:42:06.684739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.684636    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-openshift-ca\") pod \"93c79bd8-12e7-4c04-9774-e2c638b2548f\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") "
Oct 30 19:42:06.684867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.684853    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-xks-ca\") pod \"93c79bd8-12e7-4c04-9774-e2c638b2548f\" (UID: \"93c79bd8-12e7-4c04-9774-e2c638b2548f\") "
Oct 30 19:42:06.685122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.685089    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-openshift-ca" pod "93c79bd8-12e7-4c04-9774-e2c638b2548f" (UID: "93c79bd8-12e7-4c04-9774-e2c638b2548f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:06.685643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.685623    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:06.686968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.686941    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c79bd8-12e7-4c04-9774-e2c638b2548f-kube-api-access-spg2k" pod "93c79bd8-12e7-4c04-9774-e2c638b2548f" (UID: "93c79bd8-12e7-4c04-9774-e2c638b2548f"). InnerVolumeSpecName "kube-api-access-spg2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:06.687067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.687009    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-xks-ca" pod "93c79bd8-12e7-4c04-9774-e2c638b2548f" (UID: "93c79bd8-12e7-4c04-9774-e2c638b2548f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:06.787366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.787336    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c79bd8-12e7-4c04-9774-e2c638b2548f-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:06.787366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.787365    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spg2k\" (UniqueName: \"kubernetes.io/projected/93c79bd8-12e7-4c04-9774-e2c638b2548f-kube-api-access-spg2k\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:06.904469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:06.904434    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:07.389670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:07.389638    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"
Oct 30 19:42:07.389670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:07.389651    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw" event={"ID":"93c79bd8-12e7-4c04-9774-e2c638b2548f","Type":"ContainerDied","Data":"bc60f3f4f8cff1a475b3b31153e34d2db5dab7734226e138c9318cf4b37e3ac1"}
Oct 30 19:42:07.389978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:07.389756    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc60f3f4f8cff1a475b3b31153e34d2db5dab7734226e138c9318cf4b37e3ac1"
Oct 30 19:42:07.550321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:07.550135    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:08.337725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:08.334246    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:09.165504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:09.165284    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:09.771710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:09.768665    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:10.530526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:10.528363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:11.295271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:11.295195    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:11.985277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:11.982043    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:12.822342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:12.822306    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:13.466715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.466206    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:13.673346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.673308    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"]
Oct 30 19:42:13.674426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.674397    4925 cpu_manager.go:414] "containerMap: removing container" podUID="93c79bd8-12e7-4c04-9774-e2c638b2548f" containerName="revoke-keys"
Oct 30 19:42:13.674426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.674424    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c79bd8-12e7-4c04-9774-e2c638b2548f" containerName="revoke-keys"
Oct 30 19:42:13.674627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.674613    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c79bd8-12e7-4c04-9774-e2c638b2548f" containerName="revoke-keys"
Oct 30 19:42:13.678528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.678510    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.683888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.683862    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:13.684011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.683973    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:42:13.686249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.686215    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"]
Oct 30 19:42:13.777540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.777450    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8c117c7-wgnh4\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.777744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.777657    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8c117c7-wgnh4\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.777952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.777932    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26dn2\" (UniqueName: \"kubernetes.io/projected/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-kube-api-access-26dn2\") pod \"maas-api-revoke-sub-a8c117c7-wgnh4\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.879434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.879408    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dn2\" (UniqueName: \"kubernetes.io/projected/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-kube-api-access-26dn2\") pod \"maas-api-revoke-sub-a8c117c7-wgnh4\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.880036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.879465    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8c117c7-wgnh4\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.880036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.879554    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8c117c7-wgnh4\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.880366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.880344    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8c117c7-wgnh4\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.882099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.882079    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8c117c7-wgnh4\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.892170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.892147    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dn2\" (UniqueName: \"kubernetes.io/projected/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-kube-api-access-26dn2\") pod \"maas-api-revoke-sub-a8c117c7-wgnh4\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:13.995074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:13.995049    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:14.250710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:14.246333    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:14.286962 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:42:14.286933    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52800a33_5c40_4d5b_aee0_2ccca0f08f1c.slice/crio-c83307a47ea8bc46cd3936b2eba746983a960368b97612073cd13bc9a4a68a5a WatchSource:0}: Error finding container c83307a47ea8bc46cd3936b2eba746983a960368b97612073cd13bc9a4a68a5a: Status 404 returned error can't find the container with id c83307a47ea8bc46cd3936b2eba746983a960368b97612073cd13bc9a4a68a5a
Oct 30 19:42:14.287425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:14.287402    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"]
Oct 30 19:42:14.504023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:14.503989    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4" event={"ID":"52800a33-5c40-4d5b-aee0-2ccca0f08f1c","Type":"ContainerStarted","Data":"c83307a47ea8bc46cd3936b2eba746983a960368b97612073cd13bc9a4a68a5a"}
Oct 30 19:42:15.064767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:15.064556    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:15.513627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:15.513591    4925 generic.go:356] "Generic (PLEG): container finished" podID="52800a33-5c40-4d5b-aee0-2ccca0f08f1c" containerID="79c9d95c1bc9c92bd675ca3c42231e7a32e34aa314703b813ab35f0f33b81d6d" exitCode=0
Oct 30 19:42:15.513821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:15.513669    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4" event={"ID":"52800a33-5c40-4d5b-aee0-2ccca0f08f1c","Type":"ContainerDied","Data":"79c9d95c1bc9c92bd675ca3c42231e7a32e34aa314703b813ab35f0f33b81d6d"}
Oct 30 19:42:15.702815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:15.701484    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:16.376178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.376134    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:16.707170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.707144    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:16.876182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.876146    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-kube-api-access-26dn2\" (UniqueName: \"kubernetes.io/projected/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-kube-api-access-26dn2\") pod \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") "
Oct 30 19:42:16.876371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.876218    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-xks-ca\") pod \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") "
Oct 30 19:42:16.876371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.876262    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-openshift-ca\") pod \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\" (UID: \"52800a33-5c40-4d5b-aee0-2ccca0f08f1c\") "
Oct 30 19:42:16.876715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.876674    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-openshift-ca" pod "52800a33-5c40-4d5b-aee0-2ccca0f08f1c" (UID: "52800a33-5c40-4d5b-aee0-2ccca0f08f1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:16.876978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.876959    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:16.878431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.878411    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-kube-api-access-26dn2" pod "52800a33-5c40-4d5b-aee0-2ccca0f08f1c" (UID: "52800a33-5c40-4d5b-aee0-2ccca0f08f1c"). InnerVolumeSpecName "kube-api-access-26dn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:16.878731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.878707    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-xks-ca" pod "52800a33-5c40-4d5b-aee0-2ccca0f08f1c" (UID: "52800a33-5c40-4d5b-aee0-2ccca0f08f1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:17.003767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.986417    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26dn2\" (UniqueName: \"kubernetes.io/projected/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-kube-api-access-26dn2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:17.003767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.986464    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52800a33-5c40-4d5b-aee0-2ccca0f08f1c-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:17.003767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:16.992847    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:17.532181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:17.532076    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4" event={"ID":"52800a33-5c40-4d5b-aee0-2ccca0f08f1c","Type":"ContainerDied","Data":"c83307a47ea8bc46cd3936b2eba746983a960368b97612073cd13bc9a4a68a5a"}
Oct 30 19:42:17.532181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:17.532141    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c83307a47ea8bc46cd3936b2eba746983a960368b97612073cd13bc9a4a68a5a"
Oct 30 19:42:17.532181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:17.532095    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"
Oct 30 19:42:17.821737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:17.821609    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:18.478739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:18.478617    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:19.081308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:19.081271    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:19.899714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:19.891270    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:20.616895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:20.616853    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:21.292996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:21.292141    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:22.083711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:22.078135    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:22.353939 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:42:22.353849    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:42:22.887262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:22.883207    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:24.193711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.192787    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.266871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.266824    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.217:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.398207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.397804    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:42:24.445600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.444793    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.473107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.470790    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.473107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.472315    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.561594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.561549    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.588093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.588053    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:42:24.589381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.589350    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded"
Oct 30 19:42:24.593784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.593750    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.644726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.644605    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:24.700125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.699037    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:24.700125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.699482    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:42:24.704763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.704730    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"]
Oct 30 19:42:24.710712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.706189    4925 cpu_manager.go:414] "containerMap: removing container" podUID="52800a33-5c40-4d5b-aee0-2ccca0f08f1c" containerName="revoke-keys"
Oct 30 19:42:24.710900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.710882    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52800a33-5c40-4d5b-aee0-2ccca0f08f1c" containerName="revoke-keys"
Oct 30 19:42:24.711205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.711189    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="52800a33-5c40-4d5b-aee0-2ccca0f08f1c" containerName="revoke-keys"
Oct 30 19:42:24.735897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.735869    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"]
Oct 30 19:42:24.736218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.736159    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:24.757761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.756282    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:24.757761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.756374    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:42:24.792865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.784147    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdwb\" (UniqueName: \"kubernetes.io/projected/caa95b6f-d809-4948-8f90-54fb7a470a77-kube-api-access-ngdwb\") pod \"maas-api-revoke-sub-356d54f2-jvw25\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:24.792865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.784224    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-356d54f2-jvw25\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:24.792865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.784277    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-356d54f2-jvw25\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:24.932522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.927948    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdwb\" (UniqueName: \"kubernetes.io/projected/caa95b6f-d809-4948-8f90-54fb7a470a77-kube-api-access-ngdwb\") pod \"maas-api-revoke-sub-356d54f2-jvw25\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:24.932522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.928027    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-356d54f2-jvw25\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:24.932522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.928075    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-356d54f2-jvw25\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:24.932522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:24.929380    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-356d54f2-jvw25\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:25.157088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:25.157055    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-356d54f2-jvw25\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:25.167933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:25.167903    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdwb\" (UniqueName: \"kubernetes.io/projected/caa95b6f-d809-4948-8f90-54fb7a470a77-kube-api-access-ngdwb\") pod \"maas-api-revoke-sub-356d54f2-jvw25\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:25.203055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:25.203024    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:25.281736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:25.270794    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.217:8000/ready\": context deadline exceeded"
Oct 30 19:42:25.286569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:25.286543    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:42:25.492199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:25.492172    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:42:25.752574 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:42:25.752547    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa95b6f_d809_4948_8f90_54fb7a470a77.slice/crio-da896dd3667b99455526e0da2112c1ef7a6d7a22ff0c9382957c0e0631ec1c01 WatchSource:0}: Error finding container da896dd3667b99455526e0da2112c1ef7a6d7a22ff0c9382957c0e0631ec1c01: Status 404 returned error can't find the container with id da896dd3667b99455526e0da2112c1ef7a6d7a22ff0c9382957c0e0631ec1c01
Oct 30 19:42:25.752919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:25.752893    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"]
Oct 30 19:42:25.820922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:25.820745    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:26.680998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:26.679882    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:26.719338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:26.719302    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25" event={"ID":"caa95b6f-d809-4948-8f90-54fb7a470a77","Type":"ContainerStarted","Data":"75feea487f5696f0ff344459c293d6800ce920baae130083b12cc0018cd21133"}
Oct 30 19:42:26.719514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:26.719394    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25" event={"ID":"caa95b6f-d809-4948-8f90-54fb7a470a77","Type":"ContainerStarted","Data":"da896dd3667b99455526e0da2112c1ef7a6d7a22ff0c9382957c0e0631ec1c01"}
Oct 30 19:42:26.740442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:26.740385    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25" podStartSLOduration=2.740370494 podStartE2EDuration="2.740370494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:26.739478046 +0000 UTC m=+2181.642491083" watchObservedRunningTime="2026-10-30 19:42:26.740370494 +0000 UTC m=+2181.643383530"
Oct 30 19:42:27.503711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:27.503251    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:27.728863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:27.728830    4925 generic.go:356] "Generic (PLEG): container finished" podID="caa95b6f-d809-4948-8f90-54fb7a470a77" containerID="75feea487f5696f0ff344459c293d6800ce920baae130083b12cc0018cd21133" exitCode=0
Oct 30 19:42:27.729309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:27.728914    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25" event={"ID":"caa95b6f-d809-4948-8f90-54fb7a470a77","Type":"ContainerDied","Data":"75feea487f5696f0ff344459c293d6800ce920baae130083b12cc0018cd21133"}
Oct 30 19:42:28.596351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:28.596314    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:28.925297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:28.925274    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:29.088320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.088273    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa95b6f-d809-4948-8f90-54fb7a470a77-kube-api-access-ngdwb\" (UniqueName: \"kubernetes.io/projected/caa95b6f-d809-4948-8f90-54fb7a470a77-kube-api-access-ngdwb\") pod \"caa95b6f-d809-4948-8f90-54fb7a470a77\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") "
Oct 30 19:42:29.088621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.088599    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-xks-ca\") pod \"caa95b6f-d809-4948-8f90-54fb7a470a77\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") "
Oct 30 19:42:29.088748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.088730    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-openshift-ca\") pod \"caa95b6f-d809-4948-8f90-54fb7a470a77\" (UID: \"caa95b6f-d809-4948-8f90-54fb7a470a77\") "
Oct 30 19:42:29.089314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.089282    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-openshift-ca" pod "caa95b6f-d809-4948-8f90-54fb7a470a77" (UID: "caa95b6f-d809-4948-8f90-54fb7a470a77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:29.089600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.089575    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.090851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.090822    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa95b6f-d809-4948-8f90-54fb7a470a77-kube-api-access-ngdwb" pod "caa95b6f-d809-4948-8f90-54fb7a470a77" (UID: "caa95b6f-d809-4948-8f90-54fb7a470a77"). InnerVolumeSpecName "kube-api-access-ngdwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:29.091284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.091266    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-xks-ca" pod "caa95b6f-d809-4948-8f90-54fb7a470a77" (UID: "caa95b6f-d809-4948-8f90-54fb7a470a77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:29.191648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.191617    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa95b6f-d809-4948-8f90-54fb7a470a77-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.191648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.191647    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngdwb\" (UniqueName: \"kubernetes.io/projected/caa95b6f-d809-4948-8f90-54fb7a470a77-kube-api-access-ngdwb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.310717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.302246    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:29.747939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.747905    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"
Oct 30 19:42:29.747939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.747935    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25" event={"ID":"caa95b6f-d809-4948-8f90-54fb7a470a77","Type":"ContainerDied","Data":"da896dd3667b99455526e0da2112c1ef7a6d7a22ff0c9382957c0e0631ec1c01"}
Oct 30 19:42:29.748204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:29.747985    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da896dd3667b99455526e0da2112c1ef7a6d7a22ff0c9382957c0e0631ec1c01"
Oct 30 19:42:30.020241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.020160    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:30.200935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.200905    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"]
Oct 30 19:42:30.201629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.201615    4925 cpu_manager.go:414] "containerMap: removing container" podUID="caa95b6f-d809-4948-8f90-54fb7a470a77" containerName="revoke-keys"
Oct 30 19:42:30.201712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.201631    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa95b6f-d809-4948-8f90-54fb7a470a77" containerName="revoke-keys"
Oct 30 19:42:30.201802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.201791    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa95b6f-d809-4948-8f90-54fb7a470a77" containerName="revoke-keys"
Oct 30 19:42:30.204345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.204326    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.209323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.209298    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:42:30.209455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.209337    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:30.213732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.213399    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"]
Oct 30 19:42:30.320529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.320441    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q27g\" (UniqueName: \"kubernetes.io/projected/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-kube-api-access-6q27g\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.320756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.320611    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.320856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.320761    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.422666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.422630    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q27g\" (UniqueName: \"kubernetes.io/projected/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-kube-api-access-6q27g\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.422889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.422816    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.422965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.422924    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.423502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.423481    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.425482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.425465    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.440930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.440909    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q27g\" (UniqueName: \"kubernetes.io/projected/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-kube-api-access-6q27g\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.524933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.524910    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:30.821433 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:42:30.821408    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ad605f_fb59_4ad1_a9b0_d144e734a76f.slice/crio-75c3da170ef4f60b76be59cfa5b1f3a3755d5a6511a53c8b28a97d5414e16fea WatchSource:0}: Error finding container 75c3da170ef4f60b76be59cfa5b1f3a3755d5a6511a53c8b28a97d5414e16fea: Status 404 returned error can't find the container with id 75c3da170ef4f60b76be59cfa5b1f3a3755d5a6511a53c8b28a97d5414e16fea
Oct 30 19:42:30.822008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:30.821983    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"]
Oct 30 19:42:31.003195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:31.002300    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:31.683413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:31.683369    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:31.765478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:31.765438    4925 generic.go:356] "Generic (PLEG): container finished" podID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerID="28dcead279e8ea1af4ba73d535bef870e04bb7d449f2d7a3465ab41e8802c13b" exitCode=7
Oct 30 19:42:31.765649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:31.765487    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" event={"ID":"f6ad605f-fb59-4ad1-a9b0-d144e734a76f","Type":"ContainerDied","Data":"28dcead279e8ea1af4ba73d535bef870e04bb7d449f2d7a3465ab41e8802c13b"}
Oct 30 19:42:31.765649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:31.765536    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" event={"ID":"f6ad605f-fb59-4ad1-a9b0-d144e734a76f","Type":"ContainerStarted","Data":"75c3da170ef4f60b76be59cfa5b1f3a3755d5a6511a53c8b28a97d5414e16fea"}
Oct 30 19:42:31.765842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:31.765828    4925 scope.go:122] "RemoveContainer" containerID="28dcead279e8ea1af4ba73d535bef870e04bb7d449f2d7a3465ab41e8802c13b"
Oct 30 19:42:32.477711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:32.476319    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:32.776849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:32.776776    4925 generic.go:356] "Generic (PLEG): container finished" podID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerID="aa69b4dbf1a73b81d2918d5cb3463c1d1e7c5fa4ab27d618482e976fa9411fb1" exitCode=7
Oct 30 19:42:32.777287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:32.776852    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" event={"ID":"f6ad605f-fb59-4ad1-a9b0-d144e734a76f","Type":"ContainerDied","Data":"aa69b4dbf1a73b81d2918d5cb3463c1d1e7c5fa4ab27d618482e976fa9411fb1"}
Oct 30 19:42:32.777287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:32.776912    4925 scope.go:122] "RemoveContainer" containerID="28dcead279e8ea1af4ba73d535bef870e04bb7d449f2d7a3465ab41e8802c13b"
Oct 30 19:42:32.777287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:32.777237    4925 scope.go:122] "RemoveContainer" containerID="aa69b4dbf1a73b81d2918d5cb3463c1d1e7c5fa4ab27d618482e976fa9411fb1"
Oct 30 19:42:32.777628 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:42:32.777602    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5_odh-ai-gateway-infra(f6ad605f-fb59-4ad1-a9b0-d144e734a76f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f"
Oct 30 19:42:33.445712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:33.445620    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:33.788179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:33.788087    4925 scope.go:122] "RemoveContainer" containerID="aa69b4dbf1a73b81d2918d5cb3463c1d1e7c5fa4ab27d618482e976fa9411fb1"
Oct 30 19:42:33.788582 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:42:33.788413    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5_odh-ai-gateway-infra(f6ad605f-fb59-4ad1-a9b0-d144e734a76f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f"
Oct 30 19:42:34.066641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:34.066551    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:34.675937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:34.675902    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:35.405789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:35.399827    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:35.696314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:35.696226    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:36.575898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:36.575862    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:37.313536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:37.311901    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:38.307150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:38.307108    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:39.360732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:39.353300    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:40.083892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:40.083853    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:40.964766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:40.964732    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:41.690732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:41.690641    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:42.399738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:42.389473    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:43.228006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:43.227967    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:43.851722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:43.847671    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:44.574079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:44.574042    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:45.345792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:45.345758    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:45.452660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:45.452626    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"]
Oct 30 19:42:45.598031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:45.585673    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:46.256032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:46.255852    4925 scope.go:122] "RemoveContainer" containerID="aa69b4dbf1a73b81d2918d5cb3463c1d1e7c5fa4ab27d618482e976fa9411fb1"
Oct 30 19:42:46.545647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:46.545564    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:46.906668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:46.906633    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" event={"ID":"f6ad605f-fb59-4ad1-a9b0-d144e734a76f","Type":"ContainerStarted","Data":"0cea377dda0c7da23f2701a23b9a2d9760c173da548059313e19b3d5b86d47ef"}
Oct 30 19:42:46.927042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:46.926994    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" podStartSLOduration=16.926979265 podStartE2EDuration="16.926979265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:46.924190556 +0000 UTC m=+2201.827203593" watchObservedRunningTime="2026-10-30 19:42:46.926979265 +0000 UTC m=+2201.829992299"
Oct 30 19:42:47.452976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:47.452883    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:47.916523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:47.916493    4925 generic.go:356] "Generic (PLEG): container finished" podID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerID="0cea377dda0c7da23f2701a23b9a2d9760c173da548059313e19b3d5b86d47ef" exitCode=7
Oct 30 19:42:47.916936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:47.916568    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" event={"ID":"f6ad605f-fb59-4ad1-a9b0-d144e734a76f","Type":"ContainerDied","Data":"0cea377dda0c7da23f2701a23b9a2d9760c173da548059313e19b3d5b86d47ef"}
Oct 30 19:42:47.916936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:47.916631    4925 scope.go:122] "RemoveContainer" containerID="aa69b4dbf1a73b81d2918d5cb3463c1d1e7c5fa4ab27d618482e976fa9411fb1"
Oct 30 19:42:47.917102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:47.917086    4925 scope.go:122] "RemoveContainer" containerID="0cea377dda0c7da23f2701a23b9a2d9760c173da548059313e19b3d5b86d47ef"
Oct 30 19:42:47.917481 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:42:47.917461    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5_odh-ai-gateway-infra(f6ad605f-fb59-4ad1-a9b0-d144e734a76f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f"
Oct 30 19:42:47.948869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:47.948841    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"]
Oct 30 19:42:48.371785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:48.370078    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:49.115017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.114993    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:49.276767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.276266    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-kube-api-access-6q27g\" (UniqueName: \"kubernetes.io/projected/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-kube-api-access-6q27g\") pod \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") "
Oct 30 19:42:49.276767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.276548    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-openshift-ca\") pod \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") "
Oct 30 19:42:49.277058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.276765    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-xks-ca\") pod \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\" (UID: \"f6ad605f-fb59-4ad1-a9b0-d144e734a76f\") "
Oct 30 19:42:49.286423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.286391    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:49.288359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.288332    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-openshift-ca" pod "f6ad605f-fb59-4ad1-a9b0-d144e734a76f" (UID: "f6ad605f-fb59-4ad1-a9b0-d144e734a76f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:49.308914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.302513    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-kube-api-access-6q27g" pod "f6ad605f-fb59-4ad1-a9b0-d144e734a76f" (UID: "f6ad605f-fb59-4ad1-a9b0-d144e734a76f"). InnerVolumeSpecName "kube-api-access-6q27g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:49.308914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.304058    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-xks-ca" pod "f6ad605f-fb59-4ad1-a9b0-d144e734a76f" (UID: "f6ad605f-fb59-4ad1-a9b0-d144e734a76f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:49.379876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.379848    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:49.379876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.379871    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:49.379876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.379887    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q27g\" (UniqueName: \"kubernetes.io/projected/f6ad605f-fb59-4ad1-a9b0-d144e734a76f-kube-api-access-6q27g\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:49.948069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.948039    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"
Oct 30 19:42:49.948279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.948080    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5" event={"ID":"f6ad605f-fb59-4ad1-a9b0-d144e734a76f","Type":"ContainerDied","Data":"75c3da170ef4f60b76be59cfa5b1f3a3755d5a6511a53c8b28a97d5414e16fea"}
Oct 30 19:42:49.948279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:49.948177    4925 scope.go:122] "RemoveContainer" containerID="0cea377dda0c7da23f2701a23b9a2d9760c173da548059313e19b3d5b86d47ef"
Oct 30 19:42:50.045570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:50.045528    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"]
Oct 30 19:42:50.050523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:50.050495    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-hsrm5"]
Oct 30 19:42:50.140355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:50.140319    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:50.777407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:50.777374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:51.266329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:51.266296    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" path="/var/lib/kubelet/pods/f6ad605f-fb59-4ad1-a9b0-d144e734a76f/volumes"
Oct 30 19:42:51.661615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:51.661583    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:52.129832 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:42:52.129742    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:42:52.336929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:52.336892    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:53.127403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:53.127366    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:53.823833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:53.823798    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:54.459375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:54.459329    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:55.089403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:55.089363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:55.853749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:55.853710    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:56.186378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:56.186350    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:56.362715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:56.362640    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:56.854309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:56.854274    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:57.564354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:57.564321    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:58.083074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:58.083046    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:58.365784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:58.364867    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:58.974714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:58.973789    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:59.604433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:42:59.604396    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:00.391143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.391107    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:00.475874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.475839    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerName="main" containerID="cri-o://e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc" gracePeriod=45
Oct 30 19:43:00.766643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.766619    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:43:00.851536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.851506    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-kube-api-access-8hmrr\" (UniqueName: \"kubernetes.io/projected/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-kube-api-access-8hmrr\") pod \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") "
Oct 30 19:43:00.851804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.851579    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-home\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-home\") pod \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") "
Oct 30 19:43:00.851804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.851642    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tls-certs\" (UniqueName: \"kubernetes.io/secret/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tls-certs\") pod \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") "
Oct 30 19:43:00.851804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.851668    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tmp-dir\") pod \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") "
Oct 30 19:43:00.852005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.851927    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-dshm\") pod \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") "
Oct 30 19:43:00.852005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.851940    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-home" pod "4386bb61-9200-4a7a-a3c4-ee73bdeb831a" (UID: "4386bb61-9200-4a7a-a3c4-ee73bdeb831a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:00.852005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.851982    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tmp-dir" pod "4386bb61-9200-4a7a-a3c4-ee73bdeb831a" (UID: "4386bb61-9200-4a7a-a3c4-ee73bdeb831a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:00.852183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.852132    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-model-cache\") pod \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\" (UID: \"4386bb61-9200-4a7a-a3c4-ee73bdeb831a\") "
Oct 30 19:43:00.852804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.852772    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-model-cache" pod "4386bb61-9200-4a7a-a3c4-ee73bdeb831a" (UID: "4386bb61-9200-4a7a-a3c4-ee73bdeb831a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:00.853157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.853128    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:00.853252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.853162    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:00.853252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.853178    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:00.854241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.854210    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-kube-api-access-8hmrr" pod "4386bb61-9200-4a7a-a3c4-ee73bdeb831a" (UID: "4386bb61-9200-4a7a-a3c4-ee73bdeb831a"). InnerVolumeSpecName "kube-api-access-8hmrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:00.854366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.854342    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tls-certs" pod "4386bb61-9200-4a7a-a3c4-ee73bdeb831a" (UID: "4386bb61-9200-4a7a-a3c4-ee73bdeb831a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:00.854543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.854521    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-dshm" pod "4386bb61-9200-4a7a-a3c4-ee73bdeb831a" (UID: "4386bb61-9200-4a7a-a3c4-ee73bdeb831a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:00.949813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.948375    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:00.968869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.964100    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hmrr\" (UniqueName: \"kubernetes.io/projected/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-kube-api-access-8hmrr\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:00.968869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.964139    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:00.968869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:00.964157    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4386bb61-9200-4a7a-a3c4-ee73bdeb831a-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:01.055513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.055479    4925 generic.go:356] "Generic (PLEG): container finished" podID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerID="e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc" exitCode=0
Oct 30 19:43:01.055731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.055525    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"
Oct 30 19:43:01.055731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.055568    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" event={"ID":"4386bb61-9200-4a7a-a3c4-ee73bdeb831a","Type":"ContainerDied","Data":"e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc"}
Oct 30 19:43:01.055731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.055606    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69" event={"ID":"4386bb61-9200-4a7a-a3c4-ee73bdeb831a","Type":"ContainerDied","Data":"c2f5c34a384dd131ddb66a5ab2a70a60da2fb8ee6ccad19e968871cb3cc1d88b"}
Oct 30 19:43:01.055731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.055622    4925 scope.go:122] "RemoveContainer" containerID="e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc"
Oct 30 19:43:01.067928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.067908    4925 scope.go:122] "RemoveContainer" containerID="e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc"
Oct 30 19:43:01.068194 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:01.068172    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc\": container with ID starting with e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc not found: ID does not exist" containerID="e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc"
Oct 30 19:43:01.068246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.068204    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc"} err="failed to get container status \"e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc\": rpc error: code = NotFound desc = could not find container \"e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc\": container with ID starting with e5dcbe4aa4866a00829a1ab97b9b4bb45c62f803ec325aa406fb8d4d0014fdcc not found: ID does not exist"
Oct 30 19:43:01.082542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.082511    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"]
Oct 30 19:43:01.087109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.087087    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0a6e66/sub-test-model-0a6e66-kserve-58d5697fd9-kvw69"]
Oct 30 19:43:01.265600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.265522    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" path="/var/lib/kubelet/pods/4386bb61-9200-4a7a-a3c4-ee73bdeb831a/volumes"
Oct 30 19:43:01.584779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:01.583190    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:02.370733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:02.369080    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:02.979314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:02.978206    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:03.769712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:03.767366    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:05.334713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:05.325178    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:05.999727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:05.997494    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:06.762141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:06.762104    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:07.531874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.531842    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"]
Oct 30 19:43:07.532790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.532771    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerName="revoke-keys"
Oct 30 19:43:07.532856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.532791    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerName="revoke-keys"
Oct 30 19:43:07.532856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.532822    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerName="revoke-keys"
Oct 30 19:43:07.532856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.532828    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerName="revoke-keys"
Oct 30 19:43:07.532962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.532860    4925 cpu_manager.go:414] "containerMap: removing container" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerName="main"
Oct 30 19:43:07.532962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.532865    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerName="main"
Oct 30 19:43:07.533032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.532975    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerName="revoke-keys"
Oct 30 19:43:07.533032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.532988    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4386bb61-9200-4a7a-a3c4-ee73bdeb831a" containerName="main"
Oct 30 19:43:07.533032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.532995    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerName="revoke-keys"
Oct 30 19:43:07.538107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.538090    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:07.543844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.543819    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:43:07.543995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.543884    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:07.545550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.545525    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"]
Oct 30 19:43:07.740026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.730741    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:07.758438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.758355    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8fe9f5-r6cw4\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:07.758643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.758453    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8fe9f5-r6cw4\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:07.758809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.758738    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7v9r\" (UniqueName: \"kubernetes.io/projected/dc685e1c-aa9f-4851-9257-718a48a02781-kube-api-access-j7v9r\") pod \"maas-api-revoke-sub-5f8fe9f5-r6cw4\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:07.860381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.860293    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7v9r\" (UniqueName: \"kubernetes.io/projected/dc685e1c-aa9f-4851-9257-718a48a02781-kube-api-access-j7v9r\") pod \"maas-api-revoke-sub-5f8fe9f5-r6cw4\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:07.860860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.860486    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8fe9f5-r6cw4\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:07.860860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.860512    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8fe9f5-r6cw4\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:07.861192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.861172    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8fe9f5-r6cw4\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:07.863276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.863247    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8fe9f5-r6cw4\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:07.872354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:07.872326    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7v9r\" (UniqueName: \"kubernetes.io/projected/dc685e1c-aa9f-4851-9257-718a48a02781-kube-api-access-j7v9r\") pod \"maas-api-revoke-sub-5f8fe9f5-r6cw4\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:08.155515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:08.155486    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:08.378842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:08.374831    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:08.441360 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:08.441328    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc685e1c_aa9f_4851_9257_718a48a02781.slice/crio-c45f9ad849de3defeaa50ae1a6bc75c2e24e398ba05170ba4df18164ec863500 WatchSource:0}: Error finding container c45f9ad849de3defeaa50ae1a6bc75c2e24e398ba05170ba4df18164ec863500: Status 404 returned error can't find the container with id c45f9ad849de3defeaa50ae1a6bc75c2e24e398ba05170ba4df18164ec863500
Oct 30 19:43:08.441890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:08.441802    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"]
Oct 30 19:43:09.055777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:09.051255    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:09.149677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:09.149637    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4" event={"ID":"dc685e1c-aa9f-4851-9257-718a48a02781","Type":"ContainerStarted","Data":"c3590a594be029f8bccd357b96ab2c91ca0ab0ff6622c70862e4d799655c42da"}
Oct 30 19:43:09.149902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:09.149723    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4" event={"ID":"dc685e1c-aa9f-4851-9257-718a48a02781","Type":"ContainerStarted","Data":"c45f9ad849de3defeaa50ae1a6bc75c2e24e398ba05170ba4df18164ec863500"}
Oct 30 19:43:09.170829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:09.170774    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4" podStartSLOduration=2.170761004 podStartE2EDuration="2.170761004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:09.165873449 +0000 UTC m=+2224.068886485" watchObservedRunningTime="2026-10-30 19:43:09.170761004 +0000 UTC m=+2224.073774083"
Oct 30 19:43:09.379023 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:09.378980    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc685e1c_aa9f_4851_9257_718a48a02781.slice/crio-c3590a594be029f8bccd357b96ab2c91ca0ab0ff6622c70862e4d799655c42da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc685e1c_aa9f_4851_9257_718a48a02781.slice/crio-conmon-c3590a594be029f8bccd357b96ab2c91ca0ab0ff6622c70862e4d799655c42da.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:43:09.880860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:09.880812    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:10.161268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:10.161236    4925 generic.go:356] "Generic (PLEG): container finished" podID="dc685e1c-aa9f-4851-9257-718a48a02781" containerID="c3590a594be029f8bccd357b96ab2c91ca0ab0ff6622c70862e4d799655c42da" exitCode=0
Oct 30 19:43:10.161668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:10.161318    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4" event={"ID":"dc685e1c-aa9f-4851-9257-718a48a02781","Type":"ContainerDied","Data":"c3590a594be029f8bccd357b96ab2c91ca0ab0ff6622c70862e4d799655c42da"}
Oct 30 19:43:10.199907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:10.199880    4925 scope.go:122] "RemoveContainer" containerID="4f06786c16927ed2cc66a587f33012b190499afe4fcc41ba4ef906e0bfec4c2b"
Oct 30 19:43:10.542738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:10.537511    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:11.209714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.202006    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:11.413553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.413527    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:11.574583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.574502    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-openshift-ca\") pod \"dc685e1c-aa9f-4851-9257-718a48a02781\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") "
Oct 30 19:43:11.574583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.574565    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-xks-ca\") pod \"dc685e1c-aa9f-4851-9257-718a48a02781\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") "
Oct 30 19:43:11.574856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.574643    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc685e1c-aa9f-4851-9257-718a48a02781-kube-api-access-j7v9r\" (UniqueName: \"kubernetes.io/projected/dc685e1c-aa9f-4851-9257-718a48a02781-kube-api-access-j7v9r\") pod \"dc685e1c-aa9f-4851-9257-718a48a02781\" (UID: \"dc685e1c-aa9f-4851-9257-718a48a02781\") "
Oct 30 19:43:11.574956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.574865    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-openshift-ca" pod "dc685e1c-aa9f-4851-9257-718a48a02781" (UID: "dc685e1c-aa9f-4851-9257-718a48a02781"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:11.575266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.575243    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:11.577037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.577014    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-xks-ca" pod "dc685e1c-aa9f-4851-9257-718a48a02781" (UID: "dc685e1c-aa9f-4851-9257-718a48a02781"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:11.577141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.577053    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc685e1c-aa9f-4851-9257-718a48a02781-kube-api-access-j7v9r" pod "dc685e1c-aa9f-4851-9257-718a48a02781" (UID: "dc685e1c-aa9f-4851-9257-718a48a02781"). InnerVolumeSpecName "kube-api-access-j7v9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:11.676872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.676838    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc685e1c-aa9f-4851-9257-718a48a02781-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:11.676872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.676864    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7v9r\" (UniqueName: \"kubernetes.io/projected/dc685e1c-aa9f-4851-9257-718a48a02781-kube-api-access-j7v9r\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:11.934720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:11.932370    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:12.220405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:12.220324    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"
Oct 30 19:43:12.220405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:12.220352    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4" event={"ID":"dc685e1c-aa9f-4851-9257-718a48a02781","Type":"ContainerDied","Data":"c45f9ad849de3defeaa50ae1a6bc75c2e24e398ba05170ba4df18164ec863500"}
Oct 30 19:43:12.220405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:12.220400    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45f9ad849de3defeaa50ae1a6bc75c2e24e398ba05170ba4df18164ec863500"
Oct 30 19:43:12.554988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:12.554883    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:13.176297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:13.173460    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:13.817856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:13.817818    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:13.999711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:13.999251    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:14.178711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:14.178178    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:14.435672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:14.432115    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:14.858218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:14.858116    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:15.657100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:15.657063    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:16.302714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:16.295427    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:17.186192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.186158    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:17.831487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.831455    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"]
Oct 30 19:43:17.832244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.832226    4925 cpu_manager.go:414] "containerMap: removing container" podUID="dc685e1c-aa9f-4851-9257-718a48a02781" containerName="revoke-keys"
Oct 30 19:43:17.832313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.832245    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc685e1c-aa9f-4851-9257-718a48a02781" containerName="revoke-keys"
Oct 30 19:43:17.832313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.832266    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerName="revoke-keys"
Oct 30 19:43:17.832313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.832271    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerName="revoke-keys"
Oct 30 19:43:17.832415 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.832400    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc685e1c-aa9f-4851-9257-718a48a02781" containerName="revoke-keys"
Oct 30 19:43:17.832451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.832415    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ad605f-fb59-4ad1-a9b0-d144e734a76f" containerName="revoke-keys"
Oct 30 19:43:17.836484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.836464    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:17.839820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.839801    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:17.840094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.840069    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:43:17.844129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.844108    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"]
Oct 30 19:43:17.937380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.937347    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a18d680f-sjhll\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:17.937562 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.937410    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a18d680f-sjhll\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:17.937644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:17.937619    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6q6\" (UniqueName: \"kubernetes.io/projected/64441e28-10bb-40af-8ff0-5eb837675923-kube-api-access-nb6q6\") pod \"maas-api-revoke-sub-a18d680f-sjhll\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:18.039594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.039560    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6q6\" (UniqueName: \"kubernetes.io/projected/64441e28-10bb-40af-8ff0-5eb837675923-kube-api-access-nb6q6\") pod \"maas-api-revoke-sub-a18d680f-sjhll\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:18.039842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.039784    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a18d680f-sjhll\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:18.039842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.039816    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a18d680f-sjhll\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:18.040419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.040401    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a18d680f-sjhll\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:18.042459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.042437    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a18d680f-sjhll\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:18.063513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.063483    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6q6\" (UniqueName: \"kubernetes.io/projected/64441e28-10bb-40af-8ff0-5eb837675923-kube-api-access-nb6q6\") pod \"maas-api-revoke-sub-a18d680f-sjhll\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:18.179713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.170144    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:18.179713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.170573    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:18.498463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.498436    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"]
Oct 30 19:43:18.499184 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:18.499161    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64441e28_10bb_40af_8ff0_5eb837675923.slice/crio-1b97df0e65a09a01ec4a0f01af3e5378351e8ca6634bf55f10dc74246cfc76a0 WatchSource:0}: Error finding container 1b97df0e65a09a01ec4a0f01af3e5378351e8ca6634bf55f10dc74246cfc76a0: Status 404 returned error can't find the container with id 1b97df0e65a09a01ec4a0f01af3e5378351e8ca6634bf55f10dc74246cfc76a0
Oct 30 19:43:18.861926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:18.861840    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:19.367563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:19.367527    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll" event={"ID":"64441e28-10bb-40af-8ff0-5eb837675923","Type":"ContainerStarted","Data":"79b99d254880be2c8557890b2235eb304ce610c513641554b738db6c8bdcc815"}
Oct 30 19:43:19.367761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:19.367590    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll" event={"ID":"64441e28-10bb-40af-8ff0-5eb837675923","Type":"ContainerStarted","Data":"1b97df0e65a09a01ec4a0f01af3e5378351e8ca6634bf55f10dc74246cfc76a0"}
Oct 30 19:43:19.389006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:19.388960    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll" podStartSLOduration=2.388947266 podStartE2EDuration="2.388947266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:19.384523983 +0000 UTC m=+2234.287537034" watchObservedRunningTime="2026-10-30 19:43:19.388947266 +0000 UTC m=+2234.291960325"
Oct 30 19:43:19.565077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:19.564978    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:20.223716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:20.220398    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:20.381127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:20.381095    4925 generic.go:356] "Generic (PLEG): container finished" podID="64441e28-10bb-40af-8ff0-5eb837675923" containerID="79b99d254880be2c8557890b2235eb304ce610c513641554b738db6c8bdcc815" exitCode=0
Oct 30 19:43:20.381296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:20.381160    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll" event={"ID":"64441e28-10bb-40af-8ff0-5eb837675923","Type":"ContainerDied","Data":"79b99d254880be2c8557890b2235eb304ce610c513641554b738db6c8bdcc815"}
Oct 30 19:43:21.001249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.001205    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:21.578827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.578802    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:21.747435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.747342    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-xks-ca\") pod \"64441e28-10bb-40af-8ff0-5eb837675923\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") "
Oct 30 19:43:21.747598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.747471    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-openshift-ca\") pod \"64441e28-10bb-40af-8ff0-5eb837675923\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") "
Oct 30 19:43:21.747598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.747558    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64441e28-10bb-40af-8ff0-5eb837675923-kube-api-access-nb6q6\" (UniqueName: \"kubernetes.io/projected/64441e28-10bb-40af-8ff0-5eb837675923-kube-api-access-nb6q6\") pod \"64441e28-10bb-40af-8ff0-5eb837675923\" (UID: \"64441e28-10bb-40af-8ff0-5eb837675923\") "
Oct 30 19:43:21.747953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.747908    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-openshift-ca" pod "64441e28-10bb-40af-8ff0-5eb837675923" (UID: "64441e28-10bb-40af-8ff0-5eb837675923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:21.748511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.748482    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:21.749955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.749925    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-xks-ca" pod "64441e28-10bb-40af-8ff0-5eb837675923" (UID: "64441e28-10bb-40af-8ff0-5eb837675923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:21.750080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.750059    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64441e28-10bb-40af-8ff0-5eb837675923-kube-api-access-nb6q6" pod "64441e28-10bb-40af-8ff0-5eb837675923" (UID: "64441e28-10bb-40af-8ff0-5eb837675923"). InnerVolumeSpecName "kube-api-access-nb6q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:21.850535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.850471    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64441e28-10bb-40af-8ff0-5eb837675923-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:21.850535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.850506    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb6q6\" (UniqueName: \"kubernetes.io/projected/64441e28-10bb-40af-8ff0-5eb837675923-kube-api-access-nb6q6\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:21.904447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:21.904332    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:22.203825 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:22.203781    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:43:22.398448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:22.398419    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"
Oct 30 19:43:22.398631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:22.398455    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll" event={"ID":"64441e28-10bb-40af-8ff0-5eb837675923","Type":"ContainerDied","Data":"1b97df0e65a09a01ec4a0f01af3e5378351e8ca6634bf55f10dc74246cfc76a0"}
Oct 30 19:43:22.398631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:22.398502    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b97df0e65a09a01ec4a0f01af3e5378351e8ca6634bf55f10dc74246cfc76a0"
Oct 30 19:43:22.764714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:22.759801    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:22.948994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:22.948961    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:23.002439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.002398    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"]
Oct 30 19:43:23.002970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.002934    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy" containerID="cri-o://e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417" gracePeriod=30
Oct 30 19:43:23.009775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.009749    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"]
Oct 30 19:43:23.010133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.010106    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy" containerID="cri-o://2513195b221f6d81a27ed894ec842e5e2b9c9bfd20f3927d095038eb2f6c209f" gracePeriod=30
Oct 30 19:43:23.037346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.037272    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"]
Oct 30 19:43:23.037814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.037784    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy" containerID="cri-o://72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d" gracePeriod=30
Oct 30 19:43:23.064100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.064060    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"]
Oct 30 19:43:23.064607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.064544    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy" containerID="cri-o://ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063" gracePeriod=30
Oct 30 19:43:23.169251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.169175    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:23.480052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.479995    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"]
Oct 30 19:43:23.480875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.480854    4925 cpu_manager.go:414] "containerMap: removing container" podUID="64441e28-10bb-40af-8ff0-5eb837675923" containerName="revoke-keys"
Oct 30 19:43:23.480875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.480874    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64441e28-10bb-40af-8ff0-5eb837675923" containerName="revoke-keys"
Oct 30 19:43:23.481066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.481042    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="64441e28-10bb-40af-8ff0-5eb837675923" containerName="revoke-keys"
Oct 30 19:43:23.484930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.484908    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.488640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.488617    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:43:23.488807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.488682    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:23.493351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.493325    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"]
Oct 30 19:43:23.504928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.504883    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv4j2\" (UniqueName: \"kubernetes.io/projected/84a509c5-198c-4532-86a7-01ee3b383480-kube-api-access-cv4j2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.505111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.505093    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.505186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.505170    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.606534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.606504    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4j2\" (UniqueName: \"kubernetes.io/projected/84a509c5-198c-4532-86a7-01ee3b383480-kube-api-access-cv4j2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.606717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.606672    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.606780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.606752    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.607316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.607291    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.609282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.609257    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.620292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.620270    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4j2\" (UniqueName: \"kubernetes.io/projected/84a509c5-198c-4532-86a7-01ee3b383480-kube-api-access-cv4j2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:23.748765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.744411    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:23.810576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:23.810545    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:24.099615 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:24.099582    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a509c5_198c_4532_86a7_01ee3b383480.slice/crio-77162d9b9cd64bd7d889dc4e9e68044d96190bace2fd55e064675d84135b3745 WatchSource:0}: Error finding container 77162d9b9cd64bd7d889dc4e9e68044d96190bace2fd55e064675d84135b3745: Status 404 returned error can't find the container with id 77162d9b9cd64bd7d889dc4e9e68044d96190bace2fd55e064675d84135b3745
Oct 30 19:43:24.099774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:24.099632    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"]
Oct 30 19:43:24.419813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:24.419771    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb" event={"ID":"84a509c5-198c-4532-86a7-01ee3b383480","Type":"ContainerStarted","Data":"77162d9b9cd64bd7d889dc4e9e68044d96190bace2fd55e064675d84135b3745"}
Oct 30 19:43:24.648717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:24.648342    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:25.404709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:25.402938    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:25.429900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:25.429866    4925 generic.go:356] "Generic (PLEG): container finished" podID="84a509c5-198c-4532-86a7-01ee3b383480" containerID="e5172923bc93581339aaf4c0744dd24c1527fab78cee829f4c0fec390c8988b5" exitCode=0
Oct 30 19:43:25.430085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:25.429947    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb" event={"ID":"84a509c5-198c-4532-86a7-01ee3b383480","Type":"ContainerDied","Data":"e5172923bc93581339aaf4c0744dd24c1527fab78cee829f4c0fec390c8988b5"}
Oct 30 19:43:26.331890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.331791    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:26.614368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.614348    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:26.711191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.711155    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-xks-ca\") pod \"84a509c5-198c-4532-86a7-01ee3b383480\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") "
Oct 30 19:43:26.711191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.711195    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-openshift-ca\") pod \"84a509c5-198c-4532-86a7-01ee3b383480\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") "
Oct 30 19:43:26.711407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.711350    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a509c5-198c-4532-86a7-01ee3b383480-kube-api-access-cv4j2\" (UniqueName: \"kubernetes.io/projected/84a509c5-198c-4532-86a7-01ee3b383480-kube-api-access-cv4j2\") pod \"84a509c5-198c-4532-86a7-01ee3b383480\" (UID: \"84a509c5-198c-4532-86a7-01ee3b383480\") "
Oct 30 19:43:26.711598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.711574    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-openshift-ca" pod "84a509c5-198c-4532-86a7-01ee3b383480" (UID: "84a509c5-198c-4532-86a7-01ee3b383480"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:26.712164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.712145    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:26.713502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.713481    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-xks-ca" pod "84a509c5-198c-4532-86a7-01ee3b383480" (UID: "84a509c5-198c-4532-86a7-01ee3b383480"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:26.713632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.713619    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a509c5-198c-4532-86a7-01ee3b383480-kube-api-access-cv4j2" pod "84a509c5-198c-4532-86a7-01ee3b383480" (UID: "84a509c5-198c-4532-86a7-01ee3b383480"). InnerVolumeSpecName "kube-api-access-cv4j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:26.813662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.813626    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a509c5-198c-4532-86a7-01ee3b383480-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:26.813662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:26.813660    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv4j2\" (UniqueName: \"kubernetes.io/projected/84a509c5-198c-4532-86a7-01ee3b383480-kube-api-access-cv4j2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:27.349328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:27.349291    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:27.448761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:27.448728    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb" event={"ID":"84a509c5-198c-4532-86a7-01ee3b383480","Type":"ContainerDied","Data":"77162d9b9cd64bd7d889dc4e9e68044d96190bace2fd55e064675d84135b3745"}
Oct 30 19:43:27.448761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:27.448760    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"
Oct 30 19:43:27.448971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:27.448781    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77162d9b9cd64bd7d889dc4e9e68044d96190bace2fd55e064675d84135b3745"
Oct 30 19:43:27.975737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:27.973781    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:28.359458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.359432    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:43:28.396443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.396414    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"]
Oct 30 19:43:28.397276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.397251    4925 cpu_manager.go:414] "containerMap: removing container" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy"
Oct 30 19:43:28.397276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.397276    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy"
Oct 30 19:43:28.397464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.397292    4925 cpu_manager.go:414] "containerMap: removing container" podUID="84a509c5-198c-4532-86a7-01ee3b383480" containerName="revoke-keys"
Oct 30 19:43:28.397464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.397297    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a509c5-198c-4532-86a7-01ee3b383480" containerName="revoke-keys"
Oct 30 19:43:28.397563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.397524    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerName="istio-proxy"
Oct 30 19:43:28.397563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.397544    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a509c5-198c-4532-86a7-01ee3b383480" containerName="revoke-keys"
Oct 30 19:43:28.401865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.401844    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.421472 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.421313    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"]
Oct 30 19:43:28.422329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.422304    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:43:28.448118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.448092    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:43:28.450852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.450823    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-podinfo\") pod \"56680a43-73b9-4d8f-865d-e112f76bfb19\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") "
Oct 30 19:43:28.450970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.450887    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56680a43-73b9-4d8f-865d-e112f76bfb19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56680a43-73b9-4d8f-865d-e112f76bfb19-istiod-ca-cert\") pod \"56680a43-73b9-4d8f-865d-e112f76bfb19\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") "
Oct 30 19:43:28.451060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.450970    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-certs\") pod \"56680a43-73b9-4d8f-865d-e112f76bfb19\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") "
Oct 30 19:43:28.451060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.451044    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-token\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-token\") pod \"56680a43-73b9-4d8f-865d-e112f76bfb19\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") "
Oct 30 19:43:28.451192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.451116    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-data\") pod \"56680a43-73b9-4d8f-865d-e112f76bfb19\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") "
Oct 30 19:43:28.451262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.451204    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-kube-api-access-fdxsj\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-kube-api-access-fdxsj\") pod \"56680a43-73b9-4d8f-865d-e112f76bfb19\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") "
Oct 30 19:43:28.451389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.451369    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-socket\") pod \"56680a43-73b9-4d8f-865d-e112f76bfb19\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") "
Oct 30 19:43:28.451507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.451469    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-envoy\") pod \"56680a43-73b9-4d8f-865d-e112f76bfb19\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") "
Oct 30 19:43:28.451507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.451492    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-certs" pod "56680a43-73b9-4d8f-865d-e112f76bfb19" (UID: "56680a43-73b9-4d8f-865d-e112f76bfb19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.451646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.451536    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-credential-socket\") pod \"56680a43-73b9-4d8f-865d-e112f76bfb19\" (UID: \"56680a43-73b9-4d8f-865d-e112f76bfb19\") "
Oct 30 19:43:28.451905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.451882    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56680a43-73b9-4d8f-865d-e112f76bfb19-istiod-ca-cert" pod "56680a43-73b9-4d8f-865d-e112f76bfb19" (UID: "56680a43-73b9-4d8f-865d-e112f76bfb19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:28.452131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.452101    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.452236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.452215    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.452330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.452308    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.452400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.452378    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.452785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.452755    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.452884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.452848    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88978-804e-48e5-8a76-50588ca147ea-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.453471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.453440    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.453667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.453646    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s77x\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-kube-api-access-6s77x\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.453808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.453717    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88978-804e-48e5-8a76-50588ca147ea-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.453873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.453842    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56680a43-73b9-4d8f-865d-e112f76bfb19-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.453873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.453864    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.454382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.454113    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-credential-socket" pod "56680a43-73b9-4d8f-865d-e112f76bfb19" (UID: "56680a43-73b9-4d8f-865d-e112f76bfb19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.454512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.454489    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-socket" pod "56680a43-73b9-4d8f-865d-e112f76bfb19" (UID: "56680a43-73b9-4d8f-865d-e112f76bfb19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.456313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.455756    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-data" pod "56680a43-73b9-4d8f-865d-e112f76bfb19" (UID: "56680a43-73b9-4d8f-865d-e112f76bfb19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.458732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.457946    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-podinfo" pod "56680a43-73b9-4d8f-865d-e112f76bfb19" (UID: "56680a43-73b9-4d8f-865d-e112f76bfb19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:28.467474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.467443    4925 generic.go:356] "Generic (PLEG): container finished" podID="56680a43-73b9-4d8f-865d-e112f76bfb19" containerID="72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d" exitCode=0
Oct 30 19:43:28.467632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.467612    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" event={"ID":"56680a43-73b9-4d8f-865d-e112f76bfb19","Type":"ContainerDied","Data":"72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d"}
Oct 30 19:43:28.467757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.467667    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd" event={"ID":"56680a43-73b9-4d8f-865d-e112f76bfb19","Type":"ContainerDied","Data":"39ea6216b4ffca2f16dcbc761933db3145e4898b4fbc6f66401a6d30ae806424"}
Oct 30 19:43:28.467757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.467717    4925 scope.go:122] "RemoveContainer" containerID="72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d"
Oct 30 19:43:28.467899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.467882    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"
Oct 30 19:43:28.476835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.476782    4925 generic.go:356] "Generic (PLEG): container finished" podID="6cf1b65c-2559-4d34-9539-448e37119221" containerID="e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417" exitCode=0
Oct 30 19:43:28.476974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.476949    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"
Oct 30 19:43:28.477117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.477094    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" event={"ID":"6cf1b65c-2559-4d34-9539-448e37119221","Type":"ContainerDied","Data":"e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417"}
Oct 30 19:43:28.477217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.477147    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c" event={"ID":"6cf1b65c-2559-4d34-9539-448e37119221","Type":"ContainerDied","Data":"6140406f00c50b66e70a70fbffbd4aee78aaac1cae5181edac03702c19863200"}
Oct 30 19:43:28.486436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.481850    4925 generic.go:356] "Generic (PLEG): container finished" podID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerID="ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063" exitCode=0
Oct 30 19:43:28.486436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.482182    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"
Oct 30 19:43:28.486436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.482420    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" event={"ID":"cf6ec1b5-8f21-429e-a7fc-be52be068305","Type":"ContainerDied","Data":"ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063"}
Oct 30 19:43:28.486436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.482462    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm" event={"ID":"cf6ec1b5-8f21-429e-a7fc-be52be068305","Type":"ContainerDied","Data":"210463e3823a423c3f9d7804e66d15e8e45727239cb08c9eeac26aa5befc4d2c"}
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.488322    4925 generic.go:356] "Generic (PLEG): container finished" podID="137fbb43-62a3-415b-9cd9-15dad618d603" containerID="2513195b221f6d81a27ed894ec842e5e2b9c9bfd20f3927d095038eb2f6c209f" exitCode=0
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.488391    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" event={"ID":"137fbb43-62a3-415b-9cd9-15dad618d603","Type":"ContainerDied","Data":"2513195b221f6d81a27ed894ec842e5e2b9c9bfd20f3927d095038eb2f6c209f"}
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.491173    4925 scope.go:122] "RemoveContainer" containerID="72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d"
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.491198    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-kube-api-access-fdxsj" pod "56680a43-73b9-4d8f-865d-e112f76bfb19" (UID: "56680a43-73b9-4d8f-865d-e112f76bfb19"). InnerVolumeSpecName "kube-api-access-fdxsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:28.491414    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d\": container with ID starting with 72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d not found: ID does not exist" containerID="72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d"
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.491442    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d"} err="failed to get container status \"72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d\": rpc error: code = NotFound desc = could not find container \"72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d\": container with ID starting with 72efaa8c66303721330c43d358dd87630c09566d7a3e77d10aa8ce62e5e3498d not found: ID does not exist"
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.491466    4925 scope.go:122] "RemoveContainer" containerID="e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417"
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.492501    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"]
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.492907    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-envoy" pod "56680a43-73b9-4d8f-865d-e112f76bfb19" (UID: "56680a43-73b9-4d8f-865d-e112f76bfb19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.493545    4925 cpu_manager.go:414] "containerMap: removing container" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy"
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.493565    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy"
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.493627    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy"
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.493634    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy"
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.494248    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" containerName="istio-proxy"
Oct 30 19:43:28.500180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.494281    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf1b65c-2559-4d34-9539-448e37119221" containerName="istio-proxy"
Oct 30 19:43:28.506557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.506180    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.508612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.508587    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-token" pod "56680a43-73b9-4d8f-865d-e112f76bfb19" (UID: "56680a43-73b9-4d8f-865d-e112f76bfb19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.509883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.509858    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"]
Oct 30 19:43:28.518301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.518273    4925 scope.go:122] "RemoveContainer" containerID="e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417"
Oct 30 19:43:28.518700 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:28.518657    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417\": container with ID starting with e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417 not found: ID does not exist" containerID="e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417"
Oct 30 19:43:28.519003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.518894    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417"} err="failed to get container status \"e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417\": rpc error: code = NotFound desc = could not find container \"e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417\": container with ID starting with e265f890ffc757b27c596d7f9de06999084265f01d416c78713f4dca217f1417 not found: ID does not exist"
Oct 30 19:43:28.519003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.518926    4925 scope.go:122] "RemoveContainer" containerID="ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063"
Oct 30 19:43:28.521235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.520755    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"]
Oct 30 19:43:28.529224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.529202    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.531618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.531598    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"]
Oct 30 19:43:28.532420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.532399    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-178bcc-serving-cert\""
Oct 30 19:43:28.541361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.541324    4925 scope.go:122] "RemoveContainer" containerID="ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063"
Oct 30 19:43:28.541828 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:28.541804    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063\": container with ID starting with ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063 not found: ID does not exist" containerID="ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063"
Oct 30 19:43:28.541909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.541839    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063"} err="failed to get container status \"ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063\": rpc error: code = NotFound desc = could not find container \"ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063\": container with ID starting with ee4f73cbb2d737989588c3ca0076b8356664879590d1792632f9e727a23c2063 not found: ID does not exist"
Oct 30 19:43:28.554408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554376    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-socket\") pod \"6cf1b65c-2559-4d34-9539-448e37119221\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") "
Oct 30 19:43:28.554538 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554446    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-kube-api-access-f4vr5\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-kube-api-access-f4vr5\") pod \"cf6ec1b5-8f21-429e-a7fc-be52be068305\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") "
Oct 30 19:43:28.554626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554584    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-token\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-token\") pod \"cf6ec1b5-8f21-429e-a7fc-be52be068305\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") "
Oct 30 19:43:28.554751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554622    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-socket\") pod \"cf6ec1b5-8f21-429e-a7fc-be52be068305\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") "
Oct 30 19:43:28.554751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554659    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-credential-socket\") pod \"cf6ec1b5-8f21-429e-a7fc-be52be068305\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") "
Oct 30 19:43:28.554751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554730    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-data\") pod \"6cf1b65c-2559-4d34-9539-448e37119221\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") "
Oct 30 19:43:28.554961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554761    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-kube-api-access-qxd47\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-kube-api-access-qxd47\") pod \"6cf1b65c-2559-4d34-9539-448e37119221\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") "
Oct 30 19:43:28.554961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554793    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf1b65c-2559-4d34-9539-448e37119221-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf1b65c-2559-4d34-9539-448e37119221-istiod-ca-cert\") pod \"6cf1b65c-2559-4d34-9539-448e37119221\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") "
Oct 30 19:43:28.554961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554836    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6ec1b5-8f21-429e-a7fc-be52be068305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6ec1b5-8f21-429e-a7fc-be52be068305-istiod-ca-cert\") pod \"cf6ec1b5-8f21-429e-a7fc-be52be068305\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") "
Oct 30 19:43:28.555144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.554963    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-credential-socket\") pod \"6cf1b65c-2559-4d34-9539-448e37119221\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") "
Oct 30 19:43:28.555144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555089    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-certs\") pod \"cf6ec1b5-8f21-429e-a7fc-be52be068305\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") "
Oct 30 19:43:28.555144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555119    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-data\") pod \"cf6ec1b5-8f21-429e-a7fc-be52be068305\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") "
Oct 30 19:43:28.555296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555153    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-podinfo\") pod \"cf6ec1b5-8f21-429e-a7fc-be52be068305\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") "
Oct 30 19:43:28.555296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555194    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-istio-token\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-istio-token\") pod \"6cf1b65c-2559-4d34-9539-448e37119221\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") "
Oct 30 19:43:28.555296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555226    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cf1b65c-2559-4d34-9539-448e37119221-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf1b65c-2559-4d34-9539-448e37119221-istio-podinfo\") pod \"6cf1b65c-2559-4d34-9539-448e37119221\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") "
Oct 30 19:43:28.555409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555295    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-envoy\") pod \"6cf1b65c-2559-4d34-9539-448e37119221\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") "
Oct 30 19:43:28.555409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555367    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-envoy\") pod \"cf6ec1b5-8f21-429e-a7fc-be52be068305\" (UID: \"cf6ec1b5-8f21-429e-a7fc-be52be068305\") "
Oct 30 19:43:28.555485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555471    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-certs\") pod \"6cf1b65c-2559-4d34-9539-448e37119221\" (UID: \"6cf1b65c-2559-4d34-9539-448e37119221\") "
Oct 30 19:43:28.555772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555744    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-certs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.555863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555800    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.555930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555879    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.555978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555934    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.555978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.555959    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-envoy\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.556052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556015    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.556092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556056    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-istio-token\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.556129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556100    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.556174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556132    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088a7b18-ab88-4099-a815-99c6f57835ed-istiod-ca-cert\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.556231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556214    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088a7b18-ab88-4099-a815-99c6f57835ed-istio-podinfo\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.556420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556402    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.556476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556461    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88978-804e-48e5-8a76-50588ca147ea-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.556545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556531    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-data\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.556633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556617    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556760    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-credential-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556826    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mvs\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-kube-api-access-t9mvs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556909    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s77x\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-kube-api-access-6s77x\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556951    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88978-804e-48e5-8a76-50588ca147ea-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.557038    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.557057    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdxsj\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-kube-api-access-fdxsj\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.557073    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.557090    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.557105    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56680a43-73b9-4d8f-865d-e112f76bfb19-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.557119    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.557205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.557133    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56680a43-73b9-4d8f-865d-e112f76bfb19-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.560358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.560327    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.560480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.560433    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.562530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.562503    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.563053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.563000    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.563053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556012    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6ec1b5-8f21-429e-a7fc-be52be068305-istiod-ca-cert" pod "cf6ec1b5-8f21-429e-a7fc-be52be068305" (UID: "cf6ec1b5-8f21-429e-a7fc-be52be068305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.556264    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-socket" pod "6cf1b65c-2559-4d34-9539-448e37119221" (UID: "6cf1b65c-2559-4d34-9539-448e37119221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.557028    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-credential-socket" pod "cf6ec1b5-8f21-429e-a7fc-be52be068305" (UID: "cf6ec1b5-8f21-429e-a7fc-be52be068305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.557629    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf1b65c-2559-4d34-9539-448e37119221-istiod-ca-cert" pod "6cf1b65c-2559-4d34-9539-448e37119221" (UID: "6cf1b65c-2559-4d34-9539-448e37119221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.558026    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-credential-socket" pod "6cf1b65c-2559-4d34-9539-448e37119221" (UID: "6cf1b65c-2559-4d34-9539-448e37119221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.558205    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-certs" pod "cf6ec1b5-8f21-429e-a7fc-be52be068305" (UID: "cf6ec1b5-8f21-429e-a7fc-be52be068305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.558386    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-data" pod "cf6ec1b5-8f21-429e-a7fc-be52be068305" (UID: "cf6ec1b5-8f21-429e-a7fc-be52be068305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.560013    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-kube-api-access-f4vr5" pod "cf6ec1b5-8f21-429e-a7fc-be52be068305" (UID: "cf6ec1b5-8f21-429e-a7fc-be52be068305"). InnerVolumeSpecName "kube-api-access-f4vr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.561488    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-data" pod "6cf1b65c-2559-4d34-9539-448e37119221" (UID: "6cf1b65c-2559-4d34-9539-448e37119221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.561733    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-token" pod "cf6ec1b5-8f21-429e-a7fc-be52be068305" (UID: "cf6ec1b5-8f21-429e-a7fc-be52be068305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.561787    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-podinfo" pod "cf6ec1b5-8f21-429e-a7fc-be52be068305" (UID: "cf6ec1b5-8f21-429e-a7fc-be52be068305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.561803    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-certs" pod "6cf1b65c-2559-4d34-9539-448e37119221" (UID: "6cf1b65c-2559-4d34-9539-448e37119221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.563205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.561833    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-socket" pod "cf6ec1b5-8f21-429e-a7fc-be52be068305" (UID: "cf6ec1b5-8f21-429e-a7fc-be52be068305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.563973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.563454    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-kube-api-access-qxd47" pod "6cf1b65c-2559-4d34-9539-448e37119221" (UID: "6cf1b65c-2559-4d34-9539-448e37119221"). InnerVolumeSpecName "kube-api-access-qxd47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.563973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.563868    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-istio-token" pod "6cf1b65c-2559-4d34-9539-448e37119221" (UID: "6cf1b65c-2559-4d34-9539-448e37119221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.564854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.564806    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.566530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.566292    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cf1b65c-2559-4d34-9539-448e37119221-istio-podinfo" pod "6cf1b65c-2559-4d34-9539-448e37119221" (UID: "6cf1b65c-2559-4d34-9539-448e37119221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:28.566530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.566358    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-envoy" pod "cf6ec1b5-8f21-429e-a7fc-be52be068305" (UID: "cf6ec1b5-8f21-429e-a7fc-be52be068305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.567536 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.567465    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88978-804e-48e5-8a76-50588ca147ea-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.567631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.567578    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-envoy" pod "6cf1b65c-2559-4d34-9539-448e37119221" (UID: "6cf1b65c-2559-4d34-9539-448e37119221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.571249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.571222    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88978-804e-48e5-8a76-50588ca147ea-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.577322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.577278    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.604529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.604487    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s77x\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-kube-api-access-6s77x\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.659477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.659432    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-certs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.659644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.659554    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-envoy\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.659644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.659602    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.659840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.659644    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-istio-token\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.659840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.659705    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.659840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.659733    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088a7b18-ab88-4099-a815-99c6f57835ed-istiod-ca-cert\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.659840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.659795    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088a7b18-ab88-4099-a815-99c6f57835ed-istio-podinfo\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.660070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.659947    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.660070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.659973    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a41b6b-9785-472f-b815-0b6db3d7df76-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.660070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660029    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-data\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.660244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660152    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9995v\" (UniqueName: \"kubernetes.io/projected/86a41b6b-9785-472f-b815-0b6db3d7df76-kube-api-access-9995v\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.660244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660185    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-credential-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.660244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660234    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mvs\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-kube-api-access-t9mvs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.660410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660340    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660356    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660372    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660386    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660401    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxd47\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-kube-api-access-qxd47\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660416    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf1b65c-2559-4d34-9539-448e37119221-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660429    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6ec1b5-8f21-429e-a7fc-be52be068305-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660443    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660457    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660470    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660484    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660498    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf1b65c-2559-4d34-9539-448e37119221-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660511    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf1b65c-2559-4d34-9539-448e37119221-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660526    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660538    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6ec1b5-8f21-429e-a7fc-be52be068305-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660554    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660568    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf1b65c-2559-4d34-9539-448e37119221-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.660671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.660581    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4vr5\" (UniqueName: \"kubernetes.io/projected/cf6ec1b5-8f21-429e-a7fc-be52be068305-kube-api-access-f4vr5\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.661385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.661275    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-certs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.661918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.661882    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-data\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.662546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.662519    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.663359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.663335    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088a7b18-ab88-4099-a815-99c6f57835ed-istiod-ca-cert\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.663610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.663590    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-credential-socket\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.665526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.665502    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-envoy\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.668071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.668045    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088a7b18-ab88-4099-a815-99c6f57835ed-istio-podinfo\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.673939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.673913    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-istio-token\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.674208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.674190    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mvs\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-kube-api-access-t9mvs\") pod \"e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") " pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.725596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.725572    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:43:28.730263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.730243    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:28.763028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.762986    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.763224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.763037    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a41b6b-9785-472f-b815-0b6db3d7df76-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.763224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.763182    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9995v\" (UniqueName: \"kubernetes.io/projected/86a41b6b-9785-472f-b815-0b6db3d7df76-kube-api-access-9995v\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.764052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.764013    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.803781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.787358    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a41b6b-9785-472f-b815-0b6db3d7df76-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.803781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.793778    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:28.823617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.823581    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"]
Oct 30 19:43:28.824896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.824868    4925 cpu_manager.go:414] "containerMap: removing container" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy"
Oct 30 19:43:28.824896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.824895    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy"
Oct 30 19:43:28.825096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.825084    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" containerName="istio-proxy"
Oct 30 19:43:28.831447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.830889    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:28.831447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.831037    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.832723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.832679    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"]
Oct 30 19:43:28.838337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.838315    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.839059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.838796    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.841302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.841280    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9995v\" (UniqueName: \"kubernetes.io/projected/86a41b6b-9785-472f-b815-0b6db3d7df76-kube-api-access-9995v\") pod \"maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.848949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.848926    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:28.879065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.879035    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"]
Oct 30 19:43:28.883744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.883665    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-envoy\") pod \"137fbb43-62a3-415b-9cd9-15dad618d603\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") "
Oct 30 19:43:28.883880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.883849    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137fbb43-62a3-415b-9cd9-15dad618d603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137fbb43-62a3-415b-9cd9-15dad618d603-istiod-ca-cert\") pod \"137fbb43-62a3-415b-9cd9-15dad618d603\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") "
Oct 30 19:43:28.883981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.883961    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-socket\") pod \"137fbb43-62a3-415b-9cd9-15dad618d603\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") "
Oct 30 19:43:28.884063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.884028    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/137fbb43-62a3-415b-9cd9-15dad618d603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137fbb43-62a3-415b-9cd9-15dad618d603-istio-podinfo\") pod \"137fbb43-62a3-415b-9cd9-15dad618d603\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") "
Oct 30 19:43:28.884124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.884077    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-kube-api-access-j6s6s\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-kube-api-access-j6s6s\") pod \"137fbb43-62a3-415b-9cd9-15dad618d603\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") "
Oct 30 19:43:28.884181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.884142    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-istio-token\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-istio-token\") pod \"137fbb43-62a3-415b-9cd9-15dad618d603\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") "
Oct 30 19:43:28.884181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.884170    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-data\") pod \"137fbb43-62a3-415b-9cd9-15dad618d603\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") "
Oct 30 19:43:28.884306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.884225    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-certs\") pod \"137fbb43-62a3-415b-9cd9-15dad618d603\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") "
Oct 30 19:43:28.884507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.884467    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-credential-socket\") pod \"137fbb43-62a3-415b-9cd9-15dad618d603\" (UID: \"137fbb43-62a3-415b-9cd9-15dad618d603\") "
Oct 30 19:43:28.887141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.885642    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-credential-socket" pod "137fbb43-62a3-415b-9cd9-15dad618d603" (UID: "137fbb43-62a3-415b-9cd9-15dad618d603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.887141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.886036    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-data" pod "137fbb43-62a3-415b-9cd9-15dad618d603" (UID: "137fbb43-62a3-415b-9cd9-15dad618d603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.887141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.886210    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-certs" pod "137fbb43-62a3-415b-9cd9-15dad618d603" (UID: "137fbb43-62a3-415b-9cd9-15dad618d603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.887141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.886519    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137fbb43-62a3-415b-9cd9-15dad618d603-istiod-ca-cert" pod "137fbb43-62a3-415b-9cd9-15dad618d603" (UID: "137fbb43-62a3-415b-9cd9-15dad618d603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:28.891870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.891442    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-istio-token" pod "137fbb43-62a3-415b-9cd9-15dad618d603" (UID: "137fbb43-62a3-415b-9cd9-15dad618d603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.891870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.891680    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-envoy" pod "137fbb43-62a3-415b-9cd9-15dad618d603" (UID: "137fbb43-62a3-415b-9cd9-15dad618d603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.892026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.891911    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-socket" pod "137fbb43-62a3-415b-9cd9-15dad618d603" (UID: "137fbb43-62a3-415b-9cd9-15dad618d603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.899932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.899904    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-w57dm"]
Oct 30 19:43:28.903234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.903210    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-kube-api-access-j6s6s" pod "137fbb43-62a3-415b-9cd9-15dad618d603" (UID: "137fbb43-62a3-415b-9cd9-15dad618d603"). InnerVolumeSpecName "kube-api-access-j6s6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.903562 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.903534    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/137fbb43-62a3-415b-9cd9-15dad618d603-istio-podinfo" pod "137fbb43-62a3-415b-9cd9-15dad618d603" (UID: "137fbb43-62a3-415b-9cd9-15dad618d603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:28.906764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.906740    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"]
Oct 30 19:43:28.913818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.913795    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-gh54c"]
Oct 30 19:43:28.936024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.935892    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"]
Oct 30 19:43:28.936024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.935950    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-ns2qd"]
Oct 30 19:43:28.986978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.986946    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.987332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987045    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.987332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987164    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.987332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987223    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.987332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987243    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89633c90-f3a2-4de0-8b4c-306fabec3394-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.987467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987351    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.987467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987382    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.987467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987414    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqk5\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-kube-api-access-jcqk5\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.987566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987470    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:28.987566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987546    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.987566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987558    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.987663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987567    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137fbb43-62a3-415b-9cd9-15dad618d603-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.987663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987575    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.987663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987583    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137fbb43-62a3-415b-9cd9-15dad618d603-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.987663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987592    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6s6s\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-kube-api-access-j6s6s\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.987663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987602    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137fbb43-62a3-415b-9cd9-15dad618d603-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.987663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987612    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.987663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:28.987620    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137fbb43-62a3-415b-9cd9-15dad618d603-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:29.089593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.089564    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.089768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.089635    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.089768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.089660    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89633c90-f3a2-4de0-8b4c-306fabec3394-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.089907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.089829    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.089907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.089878    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.090010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.089941    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqk5\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-kube-api-access-jcqk5\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.090325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.090264    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-certs\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.090325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.090290    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-data\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.090325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.090291    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.090575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.090429    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.090575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.090507    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.090817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.090792    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89633c90-f3a2-4de0-8b4c-306fabec3394-istiod-ca-cert\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.090912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.090839    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.091087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.091066    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-credential-socket\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.092341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.092307    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-envoy\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.092787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.092767    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-podinfo\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.101661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.101640    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqk5\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-kube-api-access-jcqk5\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.101824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.101803    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-token\") pod \"e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") " pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.155668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.154868    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"]
Oct 30 19:43:29.156602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.156546    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:29.157012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.156979    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:29.157235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.157209    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:29.169123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.169097    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:29.195299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.195274    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"]
Oct 30 19:43:29.199997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.199975    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"]
Oct 30 19:43:29.202591 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:29.202561    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a41b6b_9785_472f_b815_0b6db3d7df76.slice/crio-7c3177793ee33ff7438908352fea55008d1663ef943299121eb696e0175216fb WatchSource:0}: Error finding container 7c3177793ee33ff7438908352fea55008d1663ef943299121eb696e0175216fb: Status 404 returned error can't find the container with id 7c3177793ee33ff7438908352fea55008d1663ef943299121eb696e0175216fb
Oct 30 19:43:29.202906 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:29.202880    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088a7b18_ab88_4099_a815_99c6f57835ed.slice/crio-0188eafa4d17cf4516f953529ad09740c6d228e77f0f8ba26b270ce6341e03bf WatchSource:0}: Error finding container 0188eafa4d17cf4516f953529ad09740c6d228e77f0f8ba26b270ce6341e03bf: Status 404 returned error can't find the container with id 0188eafa4d17cf4516f953529ad09740c6d228e77f0f8ba26b270ce6341e03bf
Oct 30 19:43:29.207141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.207102    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:29.207243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.207190    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:29.207243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.207236    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:29.279268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.279247    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56680a43-73b9-4d8f-865d-e112f76bfb19" path="/var/lib/kubelet/pods/56680a43-73b9-4d8f-865d-e112f76bfb19/volumes"
Oct 30 19:43:29.280157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.280139    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf1b65c-2559-4d34-9539-448e37119221" path="/var/lib/kubelet/pods/6cf1b65c-2559-4d34-9539-448e37119221/volumes"
Oct 30 19:43:29.280865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.280850    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6ec1b5-8f21-429e-a7fc-be52be068305" path="/var/lib/kubelet/pods/cf6ec1b5-8f21-429e-a7fc-be52be068305/volumes"
Oct 30 19:43:29.417714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.410576    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:29.507110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.507074    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" event={"ID":"dbd88978-804e-48e5-8a76-50588ca147ea","Type":"ContainerStarted","Data":"eeaea7a39894bf375740a962173df0edaf6e23e689a1f9b64df6f93801c4878b"}
Oct 30 19:43:29.507391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.507371    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" event={"ID":"dbd88978-804e-48e5-8a76-50588ca147ea","Type":"ContainerStarted","Data":"5cfac0932b568117d40a61bf8d0743b5480113a076cfaebabaa852c96adc1ede"}
Oct 30 19:43:29.512097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.512074    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"]
Oct 30 19:43:29.517038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.516994    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" event={"ID":"088a7b18-ab88-4099-a815-99c6f57835ed","Type":"ContainerStarted","Data":"2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f"}
Oct 30 19:43:29.517038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.517038    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" event={"ID":"088a7b18-ab88-4099-a815-99c6f57835ed","Type":"ContainerStarted","Data":"0188eafa4d17cf4516f953529ad09740c6d228e77f0f8ba26b270ce6341e03bf"}
Oct 30 19:43:29.520663 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:29.520633    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89633c90_f3a2_4de0_8b4c_306fabec3394.slice/crio-90ffeeea3c29a8cb1e55cc39729d2166f420c89665eb2bdbb988181caa8d8ca5 WatchSource:0}: Error finding container 90ffeeea3c29a8cb1e55cc39729d2166f420c89665eb2bdbb988181caa8d8ca5: Status 404 returned error can't find the container with id 90ffeeea3c29a8cb1e55cc39729d2166f420c89665eb2bdbb988181caa8d8ca5
Oct 30 19:43:29.520914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.520824    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"
Oct 30 19:43:29.520914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.520873    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw" event={"ID":"137fbb43-62a3-415b-9cd9-15dad618d603","Type":"ContainerDied","Data":"4309beefbee1af7cbb0ae997082c908df7478329412a83c37416b05914690b06"}
Oct 30 19:43:29.521078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.520915    4925 scope.go:122] "RemoveContainer" containerID="2513195b221f6d81a27ed894ec842e5e2b9c9bfd20f3927d095038eb2f6c209f"
Oct 30 19:43:29.526327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.526307    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh" event={"ID":"86a41b6b-9785-472f-b815-0b6db3d7df76","Type":"ContainerStarted","Data":"7c3177793ee33ff7438908352fea55008d1663ef943299121eb696e0175216fb"}
Oct 30 19:43:29.526849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.526817    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:29.526948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.526892    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:29.526948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.526924    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:29.535991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.535926    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podStartSLOduration=151.53591407 podStartE2EDuration="2m31.53591407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:29.531596968 +0000 UTC m=+2244.434610007" watchObservedRunningTime="2026-10-30 19:43:29.53591407 +0000 UTC m=+2244.438927104"
Oct 30 19:43:29.559043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.559022    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"]
Oct 30 19:43:29.567212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.567177    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-69c573-openshift-default-95546bd69-49cqw"]
Oct 30 19:43:29.583933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.581561    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podStartSLOduration=150.581547478 podStartE2EDuration="2m30.581547478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:29.575899278 +0000 UTC m=+2244.478912326" watchObservedRunningTime="2026-10-30 19:43:29.581547478 +0000 UTC m=+2244.484560514"
Oct 30 19:43:29.731209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.731176    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:29.774824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.774788    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"]
Oct 30 19:43:29.836133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.836095    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:29.868775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:29.868725    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:30.036044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.035949    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:30.309046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.308966    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"]
Oct 30 19:43:30.391017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.390985    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:30.630097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.607179    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh" event={"ID":"86a41b6b-9785-472f-b815-0b6db3d7df76","Type":"ContainerStarted","Data":"c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01"}
Oct 30 19:43:30.710853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.710816    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:30.711194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.711149    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" event={"ID":"89633c90-f3a2-4de0-8b4c-306fabec3394","Type":"ContainerStarted","Data":"fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986"}
Oct 30 19:43:30.711358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.711342    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" event={"ID":"89633c90-f3a2-4de0-8b4c-306fabec3394","Type":"ContainerStarted","Data":"90ffeeea3c29a8cb1e55cc39729d2166f420c89665eb2bdbb988181caa8d8ca5"}
Oct 30 19:43:30.711557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.684403    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:30.787708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.785335    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:43:30.787708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.785388    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 19:43:30.817588 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.815775    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh" podStartSLOduration=283.395069355 podStartE2EDuration="4m43.815752387s" totalImagesPullingTime="420.683032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:30.730058351 +0000 UTC m=+2245.633071389" watchObservedRunningTime="2026-10-30 19:43:30.815752387 +0000 UTC m=+2245.718765425"
Oct 30 19:43:30.880897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.880794    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:43:30.881166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.881134    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:43:30.913712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:30.911788    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podStartSLOduration=290.911752594 podStartE2EDuration="4m50.911752594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:30.779938378 +0000 UTC m=+2245.682951413" watchObservedRunningTime="2026-10-30 19:43:30.911752594 +0000 UTC m=+2245.814765631"
Oct 30 19:43:31.171577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:31.171544    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:31.172228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:31.172202    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:43:31.172397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:31.172375    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Oct 30 19:43:31.372711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:31.358854    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137fbb43-62a3-415b-9cd9-15dad618d603" path="/var/lib/kubelet/pods/137fbb43-62a3-415b-9cd9-15dad618d603/volumes"
Oct 30 19:43:31.399068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:31.399029    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:31.761713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:31.741042    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:43:31.761713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:31.741099    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 19:43:31.833573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:31.833537    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:43:31.833879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:31.833851    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:43:32.030519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:32.030431    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:32.181715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:32.172108    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:43:32.181715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:32.172170    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Oct 30 19:43:32.752581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:32.752550    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:43:32.752873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:32.752844    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 19:43:32.833189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:32.833150    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:43:32.833514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:32.833484    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:43:32.935952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:32.935918    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:33.170382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:33.170347    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:43:33.170733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:33.170675    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Oct 30 19:43:33.680710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:33.656283    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"]
Oct 30 19:43:33.680710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:33.680272    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"]
Oct 30 19:43:33.793974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:33.793932    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:33.877891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:33.823053    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:43:33.878180 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:33.878151    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 19:43:33.891036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:33.847672    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:43:33.891370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:33.891335    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:43:34.162866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.137599    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:34.187710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.173099    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:43:34.188053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.188017    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Oct 30 19:43:34.510740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.510619    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:34.699724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.699142    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:34.732714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.731775    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:43:34.732714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.731837    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 19:43:34.780776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.779789    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4" podUID="90535d48-03f0-4a14-b3ec-7fc7cd3f9e33" containerName="payload-pre-processing" containerID="cri-o://554a07afe33962d3b38c44522524ed78bfdf54a5cd4179d0ea7783a247f6180d" gracePeriod=25
Oct 30 19:43:34.845891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.833016    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:43:34.845891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:34.833074    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:43:35.156773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.125782    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:35.186412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.184065    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:43:35.186412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.184122    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Oct 30 19:43:35.201976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.201934    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:35.341961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.341926    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:35.638778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.627786    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:35.806434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.806394    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:43:36.028919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.923886    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 19:43:36.028919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.996498    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:43:36.028919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.996569    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:43:36.028919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:35.923610    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"]
Oct 30 19:43:36.136888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.136844    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:36.366260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.192256    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:36.366260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.193880    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:36.366260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.218144    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:36.366260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.354114    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:43:36.366260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.225133    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:43:36.411186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.232834    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:43:36.411489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.411456    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Oct 30 19:43:36.415716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.411778    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:43:36.419714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.417575    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:43:36.457745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.441967    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:43:36.490109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.471042    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:36.581713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.581114    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:43:36.636890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.636793    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:36.658821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.648159    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:43:36.694648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.694617    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4" event={"ID":"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33","Type":"ContainerDied","Data":"554a07afe33962d3b38c44522524ed78bfdf54a5cd4179d0ea7783a247f6180d"}
Oct 30 19:43:36.694968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.694406    4925 generic.go:356] "Generic (PLEG): container finished" podID="90535d48-03f0-4a14-b3ec-7fc7cd3f9e33" containerID="554a07afe33962d3b38c44522524ed78bfdf54a5cd4179d0ea7783a247f6180d" exitCode=0
Oct 30 19:43:36.695331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.695143    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:43:36.695814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.695530    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:37.036975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.726389    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:37.037781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.790495    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:43:37.192878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.192834    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 19:43:37.193149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.068762    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:43:37.194604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:36.822303    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:43:37.195064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.096842    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:43:37.195215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.195192    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:43:37.195328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.179754    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:43:37.195443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.195426    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Oct 30 19:43:37.203281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.149583    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:43:37.280331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.249776    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 30 19:43:37.280331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.255473    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:43:37.321771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.300414    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:37.357827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.357781    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:37.374171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.366041    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:43:37.382423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.382243    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-plugins-config-volume\") pod \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\" (UID: \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\") "
Oct 30 19:43:37.383041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.383013    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-kube-api-access-m6h6j\" (UniqueName: \"kubernetes.io/projected/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-kube-api-access-m6h6j\") pod \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\" (UID: \"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33\") "
Oct 30 19:43:37.385530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.385506    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-plugins-config-volume" pod "90535d48-03f0-4a14-b3ec-7fc7cd3f9e33" (UID: "90535d48-03f0-4a14-b3ec-7fc7cd3f9e33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:37.492902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.492262    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:37.497895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.497857    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-kube-api-access-m6h6j" pod "90535d48-03f0-4a14-b3ec-7fc7cd3f9e33" (UID: "90535d48-03f0-4a14-b3ec-7fc7cd3f9e33"). InnerVolumeSpecName "kube-api-access-m6h6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:37.552131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.500987    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:37.578593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.578509    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:37.605704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.605644    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6h6j\" (UniqueName: \"kubernetes.io/projected/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33-kube-api-access-m6h6j\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:37.665830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.665783    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:37.769836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.769798    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:43:37.770066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.769861    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 19:43:37.814044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.814002    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4" podUID="90535d48-03f0-4a14-b3ec-7fc7cd3f9e33" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.221:50052\" within 1s: context deadline exceeded"
Oct 30 19:43:37.814706 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:37.814386    4925 logging.go:55] [core] [Channel #2575 SubChannel #2576]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.221:50052", ServerName: "10.132.0.221:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.221:50052: operation was canceled"
Oct 30 19:43:37.818422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.818398    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:43:37.846108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.846025    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4" event={"ID":"90535d48-03f0-4a14-b3ec-7fc7cd3f9e33","Type":"ContainerDied","Data":"a78616b0e0bae0c186e971bb8280ad1bd29013ececa49324056db163e414d2ba"}
Oct 30 19:43:37.846386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.846362    4925 scope.go:122] "RemoveContainer" containerID="554a07afe33962d3b38c44522524ed78bfdf54a5cd4179d0ea7783a247f6180d"
Oct 30 19:43:37.846646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.829323    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:37.863913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.831594    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"
Oct 30 19:43:37.867815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.845954    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:43:37.868054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:37.868027    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:43:38.129945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.125470    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:38.129945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.126100    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:43:38.172716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.172298    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:43:38.172716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.172352    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused"
Oct 30 19:43:38.195479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.195426    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:43:38.196110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.196085    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:43:38.301789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.301749    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"]
Oct 30 19:43:38.373831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.373773    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:43:38.375291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.375253    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:38.419312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.419281    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:43:38.534322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.521082    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50648-6bcb65d87-x4jd4"]
Oct 30 19:43:38.539140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.538983    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:38.539905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.539736    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.217:8000/ready\": context deadline exceeded"
Oct 30 19:43:38.567859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.567823    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:43:38.569777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.569314    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:43:38.588743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.574459    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:38.588743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.586769    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:38.626043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.626003    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:38.644140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.643868    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:43:38.666305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.662392    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2" podUID="9abbd9ae-e288-40aa-a2ed-f12cf4073099" containerName="payload-processing" containerID="cri-o://984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53" gracePeriod=25
Oct 30 19:43:38.667551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.666724    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:43:38.680976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.680900    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt" podUID="73962804-938e-4b9c-b058-a0f071e29de5" containerName="payload-pre-processing" containerID="cri-o://bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e" gracePeriod=25
Oct 30 19:43:38.693404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.692923    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:38.763713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.760497    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:43:38.832249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.832190    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:43:38.832559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.832527    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:43:38.860118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.860092    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:43:38.885943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.882980    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:43:38.944867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.940841    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"]
Oct 30 19:43:38.944867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.942001    4925 cpu_manager.go:414] "containerMap: removing container" podUID="90535d48-03f0-4a14-b3ec-7fc7cd3f9e33" containerName="payload-pre-processing"
Oct 30 19:43:38.944867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.942024    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90535d48-03f0-4a14-b3ec-7fc7cd3f9e33" containerName="payload-pre-processing"
Oct 30 19:43:38.944867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.942302    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="90535d48-03f0-4a14-b3ec-7fc7cd3f9e33" containerName="payload-pre-processing"
Oct 30 19:43:38.949235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.949209    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:38.956829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.956665    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:43:38.957820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.957750    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:38.968489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.967999    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:38.970503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.969487    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:38.974822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.974789    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"]
Oct 30 19:43:38.977638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:38.977618    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:43:39.095309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.091330    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-547898e1-zq6kd\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.095309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.091433    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-547898e1-zq6kd\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.095309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.094180    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxr7\" (UniqueName: \"kubernetes.io/projected/cb6f42e1-b321-4c90-89a7-7a9a020189e9-kube-api-access-hzxr7\") pod \"maas-api-revoke-sub-547898e1-zq6kd\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.206196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.206095    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:39.449712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.449428    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-547898e1-zq6kd\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.449712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.449498    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-547898e1-zq6kd\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.449712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.449627    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxr7\" (UniqueName: \"kubernetes.io/projected/cb6f42e1-b321-4c90-89a7-7a9a020189e9-kube-api-access-hzxr7\") pod \"maas-api-revoke-sub-547898e1-zq6kd\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.463753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.462283    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-547898e1-zq6kd\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.486461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.486426    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90535d48-03f0-4a14-b3ec-7fc7cd3f9e33" path="/var/lib/kubelet/pods/90535d48-03f0-4a14-b3ec-7fc7cd3f9e33/volumes"
Oct 30 19:43:39.503978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.494278    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-547898e1-zq6kd\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.534517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.534486    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxr7\" (UniqueName: \"kubernetes.io/projected/cb6f42e1-b321-4c90-89a7-7a9a020189e9-kube-api-access-hzxr7\") pod \"maas-api-revoke-sub-547898e1-zq6kd\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.647552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.646881    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:39.746841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.746244    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:39.749033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.748965    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:39.760219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.756618    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:39.936983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:39.936951    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:40.014787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:40.014040    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:40.024291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:40.016251    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:40.029389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:40.027141    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:43:40.046290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:40.046238    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"]
Oct 30 19:43:40.495567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:40.495529    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:40.672713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:40.672554    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:40.920317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:40.920274    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd" event={"ID":"cb6f42e1-b321-4c90-89a7-7a9a020189e9","Type":"ContainerStarted","Data":"9d5e0441ba75da981c8f360537c1ae6c4df968e1dac071114bd0556f0467d027"}
Oct 30 19:43:40.932651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:40.932562    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd" event={"ID":"cb6f42e1-b321-4c90-89a7-7a9a020189e9","Type":"ContainerStarted","Data":"00037044499b2e2af49330d233743b354fe86d77d7024673cc21e7108c140841"}
Oct 30 19:43:41.002604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:41.002571    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:41.021146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:41.021079    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd" podStartSLOduration=3.021056532 podStartE2EDuration="3.021056532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:40.94205738 +0000 UTC m=+2255.845070418" watchObservedRunningTime="2026-10-30 19:43:41.021056532 +0000 UTC m=+2255.924069618"
Oct 30 19:43:41.030608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:41.030536    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"]
Oct 30 19:43:41.035487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:41.035461    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a6e66-mlxbb"]
Oct 30 19:43:41.189437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:41.189334    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:41.269451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:41.269411    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a509c5-198c-4532-86a7-01ee3b383480" path="/var/lib/kubelet/pods/84a509c5-198c-4532-86a7-01ee3b383480/volumes"
Oct 30 19:43:41.582015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:41.581926    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:41.930556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:41.930518    4925 generic.go:356] "Generic (PLEG): container finished" podID="cb6f42e1-b321-4c90-89a7-7a9a020189e9" containerID="9d5e0441ba75da981c8f360537c1ae6c4df968e1dac071114bd0556f0467d027" exitCode=0
Oct 30 19:43:41.930784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:41.930595    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd" event={"ID":"cb6f42e1-b321-4c90-89a7-7a9a020189e9","Type":"ContainerDied","Data":"9d5e0441ba75da981c8f360537c1ae6c4df968e1dac071114bd0556f0467d027"}
Oct 30 19:43:42.315924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:42.315777    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:42.981942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:42.981899    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt" podUID="73962804-938e-4b9c-b058-a0f071e29de5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:42.998882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:42.993803    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2" podUID="9abbd9ae-e288-40aa-a2ed-f12cf4073099" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:43.348800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.348056    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:43.488937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.488914    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:43.589454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.589425    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-xks-ca\") pod \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") "
Oct 30 19:43:43.589747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.589727    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6f42e1-b321-4c90-89a7-7a9a020189e9-kube-api-access-hzxr7\" (UniqueName: \"kubernetes.io/projected/cb6f42e1-b321-4c90-89a7-7a9a020189e9-kube-api-access-hzxr7\") pod \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") "
Oct 30 19:43:43.589865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.589826    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-openshift-ca\") pod \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\" (UID: \"cb6f42e1-b321-4c90-89a7-7a9a020189e9\") "
Oct 30 19:43:43.590179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.590150    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-openshift-ca" pod "cb6f42e1-b321-4c90-89a7-7a9a020189e9" (UID: "cb6f42e1-b321-4c90-89a7-7a9a020189e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:43.590668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.590649    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:43.591620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.591592    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-xks-ca" pod "cb6f42e1-b321-4c90-89a7-7a9a020189e9" (UID: "cb6f42e1-b321-4c90-89a7-7a9a020189e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:43.592189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.592175    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6f42e1-b321-4c90-89a7-7a9a020189e9-kube-api-access-hzxr7" pod "cb6f42e1-b321-4c90-89a7-7a9a020189e9" (UID: "cb6f42e1-b321-4c90-89a7-7a9a020189e9"). InnerVolumeSpecName "kube-api-access-hzxr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:43.692636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.692608    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6f42e1-b321-4c90-89a7-7a9a020189e9-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:43.692636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.692635    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxr7\" (UniqueName: \"kubernetes.io/projected/cb6f42e1-b321-4c90-89a7-7a9a020189e9-kube-api-access-hzxr7\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:43.987035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.987007    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"
Oct 30 19:43:43.987035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.987030    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd" event={"ID":"cb6f42e1-b321-4c90-89a7-7a9a020189e9","Type":"ContainerDied","Data":"00037044499b2e2af49330d233743b354fe86d77d7024673cc21e7108c140841"}
Oct 30 19:43:43.987439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:43.987072    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00037044499b2e2af49330d233743b354fe86d77d7024673cc21e7108c140841"
Oct 30 19:43:44.285511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:44.285397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:44.601156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:44.601069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:44.797509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:44.797476    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:45.316280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.315448    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:45.335281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.335231    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main" containerID="cri-o://5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429" gracePeriod=45
Oct 30 19:43:45.658954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.658913    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:43:45.729317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.729285    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tmp-dir\") pod \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") "
Oct 30 19:43:45.729527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.729342    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tls-certs\") pod \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") "
Oct 30 19:43:45.729527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.729458    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-kube-api-access-grk6x\" (UniqueName: \"kubernetes.io/projected/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-kube-api-access-grk6x\") pod \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") "
Oct 30 19:43:45.729527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.729493    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-dshm\") pod \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") "
Oct 30 19:43:45.729734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.729576    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-home\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-home\") pod \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") "
Oct 30 19:43:45.729734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.729587    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tmp-dir" pod "e5f698e5-7f5d-4d87-a007-c43494eb1aa4" (UID: "e5f698e5-7f5d-4d87-a007-c43494eb1aa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:45.729734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.729644    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-model-cache\") pod \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\" (UID: \"e5f698e5-7f5d-4d87-a007-c43494eb1aa4\") "
Oct 30 19:43:45.730293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.729951    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-home" pod "e5f698e5-7f5d-4d87-a007-c43494eb1aa4" (UID: "e5f698e5-7f5d-4d87-a007-c43494eb1aa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:45.730466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.730437    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-model-cache" pod "e5f698e5-7f5d-4d87-a007-c43494eb1aa4" (UID: "e5f698e5-7f5d-4d87-a007-c43494eb1aa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:45.731039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.731014    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:45.731184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.731161    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:45.731315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.731187    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:45.732171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.732139    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tls-certs" pod "e5f698e5-7f5d-4d87-a007-c43494eb1aa4" (UID: "e5f698e5-7f5d-4d87-a007-c43494eb1aa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:45.732889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.732870    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-dshm" pod "e5f698e5-7f5d-4d87-a007-c43494eb1aa4" (UID: "e5f698e5-7f5d-4d87-a007-c43494eb1aa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:45.732958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.732933    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-kube-api-access-grk6x" pod "e5f698e5-7f5d-4d87-a007-c43494eb1aa4" (UID: "e5f698e5-7f5d-4d87-a007-c43494eb1aa4"). InnerVolumeSpecName "kube-api-access-grk6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:45.768092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.768058    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:45.832860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.832832    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:45.832860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.832859    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grk6x\" (UniqueName: \"kubernetes.io/projected/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-kube-api-access-grk6x\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:45.833059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.832872    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f698e5-7f5d-4d87-a007-c43494eb1aa4-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:45.849506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.849469    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"]
Oct 30 19:43:45.850389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.850367    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main"
Oct 30 19:43:45.850389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.850388    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main"
Oct 30 19:43:45.850590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.850411    4925 cpu_manager.go:414] "containerMap: removing container" podUID="cb6f42e1-b321-4c90-89a7-7a9a020189e9" containerName="revoke-keys"
Oct 30 19:43:45.850590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.850418    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6f42e1-b321-4c90-89a7-7a9a020189e9" containerName="revoke-keys"
Oct 30 19:43:45.850747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.850670    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerName="main"
Oct 30 19:43:45.850747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.850707    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6f42e1-b321-4c90-89a7-7a9a020189e9" containerName="revoke-keys"
Oct 30 19:43:45.854938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.854916    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:45.862216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.862168    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7aa217\""
Oct 30 19:43:45.862679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.862654    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7aa217-dockercfg-wv8jw\""
Oct 30 19:43:45.864098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.863735    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"]
Oct 30 19:43:45.934601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.934455    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e287e06-4476-46bb-ae77-ee3e208f06d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8\" (UID: \"4e287e06-4476-46bb-ae77-ee3e208f06d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:45.934601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.934506    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6dpp\" (UniqueName: \"kubernetes.io/projected/4e287e06-4476-46bb-ae77-ee3e208f06d9-kube-api-access-f6dpp\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8\" (UID: \"4e287e06-4476-46bb-ae77-ee3e208f06d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:45.935868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:45.935843    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:46.004800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.004765    4925 generic.go:356] "Generic (PLEG): container finished" podID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" containerID="5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429" exitCode=0
Oct 30 19:43:46.004978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.004823    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"
Oct 30 19:43:46.004978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.004852    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" event={"ID":"e5f698e5-7f5d-4d87-a007-c43494eb1aa4","Type":"ContainerDied","Data":"5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429"}
Oct 30 19:43:46.004978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.004897    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt" event={"ID":"e5f698e5-7f5d-4d87-a007-c43494eb1aa4","Type":"ContainerDied","Data":"d7e9cbe20ad09dcd8f86122f04a09229852286644a496684d89bc91966bb4b79"}
Oct 30 19:43:46.004978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.004919    4925 scope.go:122] "RemoveContainer" containerID="5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429"
Oct 30 19:43:46.018610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.018586    4925 scope.go:122] "RemoveContainer" containerID="5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429"
Oct 30 19:43:46.018937 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:46.018918    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429\": container with ID starting with 5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429 not found: ID does not exist" containerID="5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429"
Oct 30 19:43:46.019014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.018946    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429"} err="failed to get container status \"5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429\": rpc error: code = NotFound desc = could not find container \"5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429\": container with ID starting with 5de02f4f15d8100f5aa7ed3680c08c8d1ad4c63f7123e51eef919717be33a429 not found: ID does not exist"
Oct 30 19:43:46.035232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.035209    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"]
Oct 30 19:43:46.036402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.036377    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e287e06-4476-46bb-ae77-ee3e208f06d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8\" (UID: \"4e287e06-4476-46bb-ae77-ee3e208f06d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:46.036477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.036418    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6dpp\" (UniqueName: \"kubernetes.io/projected/4e287e06-4476-46bb-ae77-ee3e208f06d9-kube-api-access-f6dpp\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8\" (UID: \"4e287e06-4476-46bb-ae77-ee3e208f06d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:46.037070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.037050    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e287e06-4476-46bb-ae77-ee3e208f06d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8\" (UID: \"4e287e06-4476-46bb-ae77-ee3e208f06d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:46.039784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.039761    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b281d5/auth-test-model-b281d5-kserve-7f968df7d9-msljt"]
Oct 30 19:43:46.051352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.051327    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6dpp\" (UniqueName: \"kubernetes.io/projected/4e287e06-4476-46bb-ae77-ee3e208f06d9-kube-api-access-f6dpp\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8\" (UID: \"4e287e06-4476-46bb-ae77-ee3e208f06d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:46.112777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.109210    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:46.176843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.175022    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:46.728875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.728839    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"]
Oct 30 19:43:46.729223 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:46.729032    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e287e06_4476_46bb_ae77_ee3e208f06d9.slice/crio-834925e8cee94a00d95d7ecc7dce521d88fe14c6c8d69d15975e5c5d821a22fc WatchSource:0}: Error finding container 834925e8cee94a00d95d7ecc7dce521d88fe14c6c8d69d15975e5c5d821a22fc: Status 404 returned error can't find the container with id 834925e8cee94a00d95d7ecc7dce521d88fe14c6c8d69d15975e5c5d821a22fc
Oct 30 19:43:46.885745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:46.885338    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:47.015854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:47.015770    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8" event={"ID":"4e287e06-4476-46bb-ae77-ee3e208f06d9","Type":"ContainerStarted","Data":"a6dedfb039eee34d7013128e29a131070473f7f5de99af18134e262e5b78a642"}
Oct 30 19:43:47.015854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:47.015820    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8" event={"ID":"4e287e06-4476-46bb-ae77-ee3e208f06d9","Type":"ContainerStarted","Data":"834925e8cee94a00d95d7ecc7dce521d88fe14c6c8d69d15975e5c5d821a22fc"}
Oct 30 19:43:47.015854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:47.015842    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:47.034978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:47.034933    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8" podStartSLOduration=2.034918193 podStartE2EDuration="2.034918193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:47.03254467 +0000 UTC m=+2261.935557707" watchObservedRunningTime="2026-10-30 19:43:47.034918193 +0000 UTC m=+2261.937931230"
Oct 30 19:43:47.275662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:47.275588    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f698e5-7f5d-4d87-a007-c43494eb1aa4" path="/var/lib/kubelet/pods/e5f698e5-7f5d-4d87-a007-c43494eb1aa4/volumes"
Oct 30 19:43:47.598771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:47.591742    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:48.358711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:48.358330    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:49.298736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.292567    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:49.325068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.325036    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"]
Oct 30 19:43:49.331613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.331583    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.335001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.334974    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:49.335157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.335091    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:43:49.337913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.337884    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"]
Oct 30 19:43:49.500144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.500105    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lc5h\" (UniqueName: \"kubernetes.io/projected/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-kube-api-access-9lc5h\") pod \"maas-api-revoke-sub-ee22e36a-v8djm\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.500628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.500159    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee22e36a-v8djm\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.500628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.500388    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee22e36a-v8djm\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.602614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.602503    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee22e36a-v8djm\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.602922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.602896    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lc5h\" (UniqueName: \"kubernetes.io/projected/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-kube-api-access-9lc5h\") pod \"maas-api-revoke-sub-ee22e36a-v8djm\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.603077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.602938    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee22e36a-v8djm\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.603332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.603307    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee22e36a-v8djm\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.606033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.606008    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee22e36a-v8djm\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.614444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.614418    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lc5h\" (UniqueName: \"kubernetes.io/projected/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-kube-api-access-9lc5h\") pod \"maas-api-revoke-sub-ee22e36a-v8djm\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.649251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.649231    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:49.892832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.892799    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:49.983718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:49.982141    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"]
Oct 30 19:43:49.996772 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:49.996735    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ae576a_0cb2_4643_a6e5_bd6460de3a26.slice/crio-4267f1284fd074abae549476d8563e029f1166abceef06930af471c6e3952c0b WatchSource:0}: Error finding container 4267f1284fd074abae549476d8563e029f1166abceef06930af471c6e3952c0b: Status 404 returned error can't find the container with id 4267f1284fd074abae549476d8563e029f1166abceef06930af471c6e3952c0b
Oct 30 19:43:50.043583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:50.043554    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm" event={"ID":"c0ae576a-0cb2-4643-a6e5-bd6460de3a26","Type":"ContainerStarted","Data":"4267f1284fd074abae549476d8563e029f1166abceef06930af471c6e3952c0b"}
Oct 30 19:43:50.065150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:50.065116    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:50.143580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:50.141760    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"]
Oct 30 19:43:50.143580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:50.142026    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy" containerID="cri-o://a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397" gracePeriod=30
Oct 30 19:43:50.153634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:50.153603    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"]
Oct 30 19:43:50.153887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:50.153859    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" containerID="cri-o://fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986" gracePeriod=30
Oct 30 19:43:50.320026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:50.319984    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:50.763314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:50.763205    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:50.930967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:50.930929    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:51.053405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:51.053372    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm" event={"ID":"c0ae576a-0cb2-4643-a6e5-bd6460de3a26","Type":"ContainerStarted","Data":"dd3ceeb3c68c6ef1c0c44c7ce099d941628d09f53d95c381e680f3b533cc7ad0"}
Oct 30 19:43:51.072064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:51.072019    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm" podStartSLOduration=2.072004999 podStartE2EDuration="2.072004999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:51.069287185 +0000 UTC m=+2265.972300221" watchObservedRunningTime="2026-10-30 19:43:51.072004999 +0000 UTC m=+2265.975018036"
Oct 30 19:43:51.174465 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:51.174433    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ae576a_0cb2_4643_a6e5_bd6460de3a26.slice/crio-dd3ceeb3c68c6ef1c0c44c7ce099d941628d09f53d95c381e680f3b533cc7ad0.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:43:51.936723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:51.936499    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:52.064010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:52.063973    4925 generic.go:356] "Generic (PLEG): container finished" podID="c0ae576a-0cb2-4643-a6e5-bd6460de3a26" containerID="dd3ceeb3c68c6ef1c0c44c7ce099d941628d09f53d95c381e680f3b533cc7ad0" exitCode=0
Oct 30 19:43:52.064195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:52.064052    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm" event={"ID":"c0ae576a-0cb2-4643-a6e5-bd6460de3a26","Type":"ContainerDied","Data":"dd3ceeb3c68c6ef1c0c44c7ce099d941628d09f53d95c381e680f3b533cc7ad0"}
Oct 30 19:43:52.130861 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:52.130825    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:43:52.532641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:52.532608    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:43:52.532833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:52.532674    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:43:52.951521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:52.951482    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2" podUID="9abbd9ae-e288-40aa-a2ed-f12cf4073099" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:52.952019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:52.951520    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt" podUID="73962804-938e-4b9c-b058-a0f071e29de5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:53.055183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.053261    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:53.270496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.270472    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:53.304297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.304267    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-xks-ca\") pod \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") "
Oct 30 19:43:53.304481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.304400    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-openshift-ca\") pod \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") "
Oct 30 19:43:53.304660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.304641    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-kube-api-access-9lc5h\" (UniqueName: \"kubernetes.io/projected/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-kube-api-access-9lc5h\") pod \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\" (UID: \"c0ae576a-0cb2-4643-a6e5-bd6460de3a26\") "
Oct 30 19:43:53.304952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.304907    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-openshift-ca" pod "c0ae576a-0cb2-4643-a6e5-bd6460de3a26" (UID: "c0ae576a-0cb2-4643-a6e5-bd6460de3a26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:53.307701 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.307299    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-xks-ca" pod "c0ae576a-0cb2-4643-a6e5-bd6460de3a26" (UID: "c0ae576a-0cb2-4643-a6e5-bd6460de3a26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:53.307701 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.307347    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.307959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.307931    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-kube-api-access-9lc5h" pod "c0ae576a-0cb2-4643-a6e5-bd6460de3a26" (UID: "c0ae576a-0cb2-4643-a6e5-bd6460de3a26"). InnerVolumeSpecName "kube-api-access-9lc5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:53.409374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.409346    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.409374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.409376    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lc5h\" (UniqueName: \"kubernetes.io/projected/c0ae576a-0cb2-4643-a6e5-bd6460de3a26-kube-api-access-9lc5h\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.419011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.418976    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main" containerID="cri-o://d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b" gracePeriod=43
Oct 30 19:43:53.472494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.472464    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.217:8000/ready\": dial tcp 10.132.0.217:8000: connect: connection refused"
Oct 30 19:43:53.701273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.701246    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:43:53.815821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.815714    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-home\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-home\") pod \"36c0c985-99bd-44b0-8869-87286a11ebb1\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") "
Oct 30 19:43:53.815988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.815946    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-dshm\") pod \"36c0c985-99bd-44b0-8869-87286a11ebb1\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") "
Oct 30 19:43:53.816038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.815989    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-model-cache\") pod \"36c0c985-99bd-44b0-8869-87286a11ebb1\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") "
Oct 30 19:43:53.816038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.816005    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-home" pod "36c0c985-99bd-44b0-8869-87286a11ebb1" (UID: "36c0c985-99bd-44b0-8869-87286a11ebb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.816127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.816101    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-tmp-dir\") pod \"36c0c985-99bd-44b0-8869-87286a11ebb1\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") "
Oct 30 19:43:53.816244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.816223    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c0c985-99bd-44b0-8869-87286a11ebb1-kube-api-access-l4hlt\" (UniqueName: \"kubernetes.io/projected/36c0c985-99bd-44b0-8869-87286a11ebb1-kube-api-access-l4hlt\") pod \"36c0c985-99bd-44b0-8869-87286a11ebb1\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") "
Oct 30 19:43:53.816244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.816235    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-model-cache" pod "36c0c985-99bd-44b0-8869-87286a11ebb1" (UID: "36c0c985-99bd-44b0-8869-87286a11ebb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.816392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.816261    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c0c985-99bd-44b0-8869-87286a11ebb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/36c0c985-99bd-44b0-8869-87286a11ebb1-tls-certs\") pod \"36c0c985-99bd-44b0-8869-87286a11ebb1\" (UID: \"36c0c985-99bd-44b0-8869-87286a11ebb1\") "
Oct 30 19:43:53.816439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.816384    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-tmp-dir" pod "36c0c985-99bd-44b0-8869-87286a11ebb1" (UID: "36c0c985-99bd-44b0-8869-87286a11ebb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.817363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.817334    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.817501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.817366    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.817501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.817380    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.818667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.818641    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-dshm" pod "36c0c985-99bd-44b0-8869-87286a11ebb1" (UID: "36c0c985-99bd-44b0-8869-87286a11ebb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.818792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.818701    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c0c985-99bd-44b0-8869-87286a11ebb1-kube-api-access-l4hlt" pod "36c0c985-99bd-44b0-8869-87286a11ebb1" (UID: "36c0c985-99bd-44b0-8869-87286a11ebb1"). InnerVolumeSpecName "kube-api-access-l4hlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:53.818792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.818731    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c0c985-99bd-44b0-8869-87286a11ebb1-tls-certs" pod "36c0c985-99bd-44b0-8869-87286a11ebb1" (UID: "36c0c985-99bd-44b0-8869-87286a11ebb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:53.924994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.920254    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:53.924994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.921956    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c0c985-99bd-44b0-8869-87286a11ebb1-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.924994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.921988    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4hlt\" (UniqueName: \"kubernetes.io/projected/36c0c985-99bd-44b0-8869-87286a11ebb1-kube-api-access-l4hlt\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.924994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:53.922005    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c0c985-99bd-44b0-8869-87286a11ebb1-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:54.095658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.095570    4925 generic.go:356] "Generic (PLEG): container finished" podID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerID="d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b" exitCode=0
Oct 30 19:43:54.095658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.095641    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"
Oct 30 19:43:54.096191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.095658    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" event={"ID":"36c0c985-99bd-44b0-8869-87286a11ebb1","Type":"ContainerDied","Data":"d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b"}
Oct 30 19:43:54.096191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.095731    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh" event={"ID":"36c0c985-99bd-44b0-8869-87286a11ebb1","Type":"ContainerDied","Data":"4946b3333546e9896741ae4e4fecff8a7902ed0693f3a4c078527bcbce9a7ef6"}
Oct 30 19:43:54.096191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.095749    4925 scope.go:122] "RemoveContainer" containerID="d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b"
Oct 30 19:43:54.097463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.097435    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm" event={"ID":"c0ae576a-0cb2-4643-a6e5-bd6460de3a26","Type":"ContainerDied","Data":"4267f1284fd074abae549476d8563e029f1166abceef06930af471c6e3952c0b"}
Oct 30 19:43:54.097463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.097462    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"
Oct 30 19:43:54.097669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.097510    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4267f1284fd074abae549476d8563e029f1166abceef06930af471c6e3952c0b"
Oct 30 19:43:54.107590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.107572    4925 scope.go:122] "RemoveContainer" containerID="d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b"
Oct 30 19:43:54.107889 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:54.107870    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b\": container with ID starting with d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b not found: ID does not exist" containerID="d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b"
Oct 30 19:43:54.107953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.107898    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b"} err="failed to get container status \"d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b\": rpc error: code = NotFound desc = could not find container \"d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b\": container with ID starting with d88e70f736a8fd4230c4786156691eb7a079d9f1269cabaae9cac95e68ea282b not found: ID does not exist"
Oct 30 19:43:54.131492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.131468    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"]
Oct 30 19:43:54.136369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.136343    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-8f3ff4/auth-test-model-8f3ff4-kserve-645f64f849-dt7hh"]
Oct 30 19:43:54.751158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.751118    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:43:54.751364 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.751177    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:43:54.868380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:54.866946    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:55.272885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.272847    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" path="/var/lib/kubelet/pods/36c0c985-99bd-44b0-8869-87286a11ebb1/volumes"
Oct 30 19:43:55.539764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.537103    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:55.557971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.557918    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:43:55.609714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.608234    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"]
Oct 30 19:43:55.609714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609409    4925 cpu_manager.go:414] "containerMap: removing container" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main"
Oct 30 19:43:55.609714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609429    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main"
Oct 30 19:43:55.609714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609519    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy"
Oct 30 19:43:55.609714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609529    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy"
Oct 30 19:43:55.609714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609580    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c0ae576a-0cb2-4643-a6e5-bd6460de3a26" containerName="revoke-keys"
Oct 30 19:43:55.609714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609589    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ae576a-0cb2-4643-a6e5-bd6460de3a26" containerName="revoke-keys"
Oct 30 19:43:55.609714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609634    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy"
Oct 30 19:43:55.609714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609642    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy"
Oct 30 19:43:55.610316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609847    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy"
Oct 30 19:43:55.610316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609865    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ae576a-0cb2-4643-a6e5-bd6460de3a26" containerName="revoke-keys"
Oct 30 19:43:55.610316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.609886    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c0c985-99bd-44b0-8869-87286a11ebb1" containerName="main"
Oct 30 19:43:55.615028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.614562    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:43:55.619380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.619348    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"]
Oct 30 19:43:55.631073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.631044    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"]
Oct 30 19:43:55.632437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.632415    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:55.662074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662038    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-credential-socket\") pod \"bc119c21-05da-46e6-a79b-f87ed81e7f95\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") "
Oct 30 19:43:55.662074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662116    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89633c90-f3a2-4de0-8b4c-306fabec3394-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89633c90-f3a2-4de0-8b4c-306fabec3394-istiod-ca-cert\") pod \"89633c90-f3a2-4de0-8b4c-306fabec3394\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") "
Oct 30 19:43:55.662471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662165    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc119c21-05da-46e6-a79b-f87ed81e7f95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc119c21-05da-46e6-a79b-f87ed81e7f95-istiod-ca-cert\") pod \"bc119c21-05da-46e6-a79b-f87ed81e7f95\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") "
Oct 30 19:43:55.662471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662223    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-envoy\") pod \"bc119c21-05da-46e6-a79b-f87ed81e7f95\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") "
Oct 30 19:43:55.662471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662301    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-data\") pod \"89633c90-f3a2-4de0-8b4c-306fabec3394\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") "
Oct 30 19:43:55.662471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662354    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-kube-api-access-jcqk5\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-kube-api-access-jcqk5\") pod \"89633c90-f3a2-4de0-8b4c-306fabec3394\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") "
Oct 30 19:43:55.662753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662536    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-socket\") pod \"89633c90-f3a2-4de0-8b4c-306fabec3394\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") "
Oct 30 19:43:55.662753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662565    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-envoy\") pod \"89633c90-f3a2-4de0-8b4c-306fabec3394\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") "
Oct 30 19:43:55.662753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662667    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-token\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-token\") pod \"bc119c21-05da-46e6-a79b-f87ed81e7f95\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") "
Oct 30 19:43:55.662753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662738    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-credential-socket\") pod \"89633c90-f3a2-4de0-8b4c-306fabec3394\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") "
Oct 30 19:43:55.662995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662800    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-data\") pod \"bc119c21-05da-46e6-a79b-f87ed81e7f95\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") "
Oct 30 19:43:55.662995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662835    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-certs\") pod \"89633c90-f3a2-4de0-8b4c-306fabec3394\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") "
Oct 30 19:43:55.662995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662889    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-podinfo\") pod \"89633c90-f3a2-4de0-8b4c-306fabec3394\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") "
Oct 30 19:43:55.662995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.662972    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-podinfo\") pod \"bc119c21-05da-46e6-a79b-f87ed81e7f95\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") "
Oct 30 19:43:55.663213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.663004    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-kube-api-access-zbg72\") pod \"bc119c21-05da-46e6-a79b-f87ed81e7f95\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") "
Oct 30 19:43:55.663213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.663053    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-certs\") pod \"bc119c21-05da-46e6-a79b-f87ed81e7f95\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") "
Oct 30 19:43:55.663213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.663115    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-token\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-token\") pod \"89633c90-f3a2-4de0-8b4c-306fabec3394\" (UID: \"89633c90-f3a2-4de0-8b4c-306fabec3394\") "
Oct 30 19:43:55.663213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.663159    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-socket\") pod \"bc119c21-05da-46e6-a79b-f87ed81e7f95\" (UID: \"bc119c21-05da-46e6-a79b-f87ed81e7f95\") "
Oct 30 19:43:55.664186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.664153    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qp9r\" (UniqueName: \"kubernetes.io/projected/d94a813f-14ef-4507-9628-2af04dd1d1fa-kube-api-access-9qp9r\") pod \"payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85\" (UID: \"d94a813f-14ef-4507-9628-2af04dd1d1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:43:55.664343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.664273    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a813f-14ef-4507-9628-2af04dd1d1fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85\" (UID: \"d94a813f-14ef-4507-9628-2af04dd1d1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.665072    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-data" pod "bc119c21-05da-46e6-a79b-f87ed81e7f95" (UID: "bc119c21-05da-46e6-a79b-f87ed81e7f95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.665405    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-credential-socket" pod "89633c90-f3a2-4de0-8b4c-306fabec3394" (UID: "89633c90-f3a2-4de0-8b4c-306fabec3394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.666529    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-data" pod "89633c90-f3a2-4de0-8b4c-306fabec3394" (UID: "89633c90-f3a2-4de0-8b4c-306fabec3394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.666851    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-socket" pod "89633c90-f3a2-4de0-8b4c-306fabec3394" (UID: "89633c90-f3a2-4de0-8b4c-306fabec3394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.667416    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-certs" pod "bc119c21-05da-46e6-a79b-f87ed81e7f95" (UID: "bc119c21-05da-46e6-a79b-f87ed81e7f95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.667729    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-socket" pod "bc119c21-05da-46e6-a79b-f87ed81e7f95" (UID: "bc119c21-05da-46e6-a79b-f87ed81e7f95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.667891    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-token" pod "bc119c21-05da-46e6-a79b-f87ed81e7f95" (UID: "bc119c21-05da-46e6-a79b-f87ed81e7f95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.668027    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-certs" pod "89633c90-f3a2-4de0-8b4c-306fabec3394" (UID: "89633c90-f3a2-4de0-8b4c-306fabec3394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.668373    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89633c90-f3a2-4de0-8b4c-306fabec3394-istiod-ca-cert" pod "89633c90-f3a2-4de0-8b4c-306fabec3394" (UID: "89633c90-f3a2-4de0-8b4c-306fabec3394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.668602    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-credential-socket" pod "bc119c21-05da-46e6-a79b-f87ed81e7f95" (UID: "bc119c21-05da-46e6-a79b-f87ed81e7f95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.668850    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc119c21-05da-46e6-a79b-f87ed81e7f95-istiod-ca-cert" pod "bc119c21-05da-46e6-a79b-f87ed81e7f95" (UID: "bc119c21-05da-46e6-a79b-f87ed81e7f95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.675191    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-token" pod "89633c90-f3a2-4de0-8b4c-306fabec3394" (UID: "89633c90-f3a2-4de0-8b4c-306fabec3394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.676942    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-podinfo" pod "bc119c21-05da-46e6-a79b-f87ed81e7f95" (UID: "bc119c21-05da-46e6-a79b-f87ed81e7f95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.676976    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-kube-api-access-jcqk5" pod "89633c90-f3a2-4de0-8b4c-306fabec3394" (UID: "89633c90-f3a2-4de0-8b4c-306fabec3394"). InnerVolumeSpecName "kube-api-access-jcqk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.676992    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-envoy" pod "bc119c21-05da-46e6-a79b-f87ed81e7f95" (UID: "bc119c21-05da-46e6-a79b-f87ed81e7f95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.679755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.677092    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-podinfo" pod "89633c90-f3a2-4de0-8b4c-306fabec3394" (UID: "89633c90-f3a2-4de0-8b4c-306fabec3394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:55.683571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.681208    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-kube-api-access-zbg72" pod "bc119c21-05da-46e6-a79b-f87ed81e7f95" (UID: "bc119c21-05da-46e6-a79b-f87ed81e7f95"). InnerVolumeSpecName "kube-api-access-zbg72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:55.683571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.681228    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-envoy" pod "89633c90-f3a2-4de0-8b4c-306fabec3394" (UID: "89633c90-f3a2-4de0-8b4c-306fabec3394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:55.705899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.705849    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e50648-dockercfg-9vfgq\""
Oct 30 19:43:55.707101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.707012    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e50648\""
Oct 30 19:43:55.723470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.723439    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"]
Oct 30 19:43:55.724549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.724521    4925 cpu_manager.go:414] "containerMap: removing container" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy"
Oct 30 19:43:55.724549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.724547    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy"
Oct 30 19:43:55.724890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.724854    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerName="istio-proxy"
Oct 30 19:43:55.724890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.724881    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerName="istio-proxy"
Oct 30 19:43:55.738596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.738564    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"]
Oct 30 19:43:55.738828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.738666    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.765993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.765938    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.766189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766006    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-data\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.766189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766034    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-token\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.766189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766142    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dc86\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-kube-api-access-5dc86\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.766495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766192    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qp9r\" (UniqueName: \"kubernetes.io/projected/d94a813f-14ef-4507-9628-2af04dd1d1fa-kube-api-access-9qp9r\") pod \"payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85\" (UID: \"d94a813f-14ef-4507-9628-2af04dd1d1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:43:55.766495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766244    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a813f-14ef-4507-9628-2af04dd1d1fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85\" (UID: \"d94a813f-14ef-4507-9628-2af04dd1d1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:43:55.766495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766274    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-credential-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.766495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766331    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-envoy\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.766495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766399    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba538c7-458e-42de-84b0-54d9e11fdcd7-istiod-ca-cert\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.766495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766428    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-certs\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.766495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766476    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-podinfo\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766743    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766762    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766778    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcqk5\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-kube-api-access-jcqk5\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766792    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766805    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766820    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766834    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766848    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766878    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89633c90-f3a2-4de0-8b4c-306fabec3394-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766895    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766910    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc119c21-05da-46e6-a79b-f87ed81e7f95-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766923    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/bc119c21-05da-46e6-a79b-f87ed81e7f95-kube-api-access-zbg72\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766935    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766948    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89633c90-f3a2-4de0-8b4c-306fabec3394-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766960    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766981    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc119c21-05da-46e6-a79b-f87ed81e7f95-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.766993    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89633c90-f3a2-4de0-8b4c-306fabec3394-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.767087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.767007    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc119c21-05da-46e6-a79b-f87ed81e7f95-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:55.768100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.767708    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a813f-14ef-4507-9628-2af04dd1d1fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85\" (UID: \"d94a813f-14ef-4507-9628-2af04dd1d1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:43:55.795873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.795821    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:55.806219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.806184    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qp9r\" (UniqueName: \"kubernetes.io/projected/d94a813f-14ef-4507-9628-2af04dd1d1fa-kube-api-access-9qp9r\") pod \"payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85\" (UID: \"d94a813f-14ef-4507-9628-2af04dd1d1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:43:55.868204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.868170    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-envoy\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.868382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.868230    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba538c7-458e-42de-84b0-54d9e11fdcd7-istiod-ca-cert\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.868382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.868359    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-certs\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.868518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.868419    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-podinfo\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.868737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.868711    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.868849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.868768    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-data\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.868849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.868770    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-certs\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.868849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.868796    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-token\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.869174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.868918    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dc86\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-kube-api-access-5dc86\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.869174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.869004    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-credential-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.869174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.869008    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.869174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.869168    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba538c7-458e-42de-84b0-54d9e11fdcd7-istiod-ca-cert\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.869399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.869385    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-data\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.869463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.869435    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-credential-socket\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.870833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.870812    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-envoy\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.871503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.871478    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-podinfo\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.884924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.884896    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-token\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.886746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.886651    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dc86\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-kube-api-access-5dc86\") pod \"e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") " pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:55.943925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:55.943892    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:43:56.059557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.059520    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:56.124557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.124520    4925 generic.go:356] "Generic (PLEG): container finished" podID="bc119c21-05da-46e6-a79b-f87ed81e7f95" containerID="a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397" exitCode=0
Oct 30 19:43:56.124740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.124614    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"
Oct 30 19:43:56.124811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.124615    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" event={"ID":"bc119c21-05da-46e6-a79b-f87ed81e7f95","Type":"ContainerDied","Data":"a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397"}
Oct 30 19:43:56.124870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.124858    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6" event={"ID":"bc119c21-05da-46e6-a79b-f87ed81e7f95","Type":"ContainerDied","Data":"2f29bab78ef5749c9d7ea7b42a53ca3e66487a58fde9a6bbe0bad783998d6f14"}
Oct 30 19:43:56.124945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.124883    4925 scope.go:122] "RemoveContainer" containerID="a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397"
Oct 30 19:43:56.127322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.127297    4925 generic.go:356] "Generic (PLEG): container finished" podID="89633c90-f3a2-4de0-8b4c-306fabec3394" containerID="fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986" exitCode=0
Oct 30 19:43:56.127438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.127409    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" event={"ID":"89633c90-f3a2-4de0-8b4c-306fabec3394","Type":"ContainerDied","Data":"fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986"}
Oct 30 19:43:56.127500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.127452    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk" event={"ID":"89633c90-f3a2-4de0-8b4c-306fabec3394","Type":"ContainerDied","Data":"90ffeeea3c29a8cb1e55cc39729d2166f420c89665eb2bdbb988181caa8d8ca5"}
Oct 30 19:43:56.127555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.127542    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"
Oct 30 19:43:56.164881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.164856    4925 scope.go:122] "RemoveContainer" containerID="40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7"
Oct 30 19:43:56.182096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.182063    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"]
Oct 30 19:43:56.188057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.188028    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-sbmhk"]
Oct 30 19:43:56.193656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.193347    4925 scope.go:122] "RemoveContainer" containerID="a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397"
Oct 30 19:43:56.194269 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:56.193914    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397\": container with ID starting with a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397 not found: ID does not exist" containerID="a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397"
Oct 30 19:43:56.194269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.193957    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397"} err="failed to get container status \"a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397\": rpc error: code = NotFound desc = could not find container \"a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397\": container with ID starting with a679d8c3e4bd01ce6399097110a66f0002b0f94befa88cf132d72b1038e1b397 not found: ID does not exist"
Oct 30 19:43:56.194269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.193981    4925 scope.go:122] "RemoveContainer" containerID="40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7"
Oct 30 19:43:56.196530 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:56.195133    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7\": container with ID starting with 40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7 not found: ID does not exist" containerID="40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7"
Oct 30 19:43:56.196530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.195161    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7"} err="failed to get container status \"40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7\": rpc error: code = NotFound desc = could not find container \"40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7\": container with ID starting with 40d5775a7dec6cfcdc871203d0c8ca71275434e4b7740780928a834e2d62abd7 not found: ID does not exist"
Oct 30 19:43:56.196530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.195180    4925 scope.go:122] "RemoveContainer" containerID="fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986"
Oct 30 19:43:56.198632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.198592    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"]
Oct 30 19:43:56.206125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.206102    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0a6e66-openshift-default-68c7bb8bdf-5xrv6"]
Oct 30 19:43:56.218525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.218501    4925 scope.go:122] "RemoveContainer" containerID="fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986"
Oct 30 19:43:56.219138 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:56.219110    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986\": container with ID starting with fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986 not found: ID does not exist" containerID="fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986"
Oct 30 19:43:56.219274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.219150    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986"} err="failed to get container status \"fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986\": rpc error: code = NotFound desc = could not find container \"fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986\": container with ID starting with fd695159e0573c4f7cc34486e5395dcbbd22018cb5355448333ac9694c4d1986 not found: ID does not exist"
Oct 30 19:43:56.339222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.336018    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"]
Oct 30 19:43:56.339222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.338073    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:56.342865 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:56.342830    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94a813f_14ef_4507_9628_2af04dd1d1fa.slice/crio-cc545b2ae2dca93f9f86eacecf14854ed9ac647830c304c0dbb6123be2ffef64 WatchSource:0}: Error finding container cc545b2ae2dca93f9f86eacecf14854ed9ac647830c304c0dbb6123be2ffef64: Status 404 returned error can't find the container with id cc545b2ae2dca93f9f86eacecf14854ed9ac647830c304c0dbb6123be2ffef64
Oct 30 19:43:56.394988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.394961    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"]
Oct 30 19:43:56.396108 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:43:56.395722    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba538c7_458e_42de_84b0_54d9e11fdcd7.slice/crio-026739227bf6dccf28b428749884ab63aa3a86be29e35ba8bf881589104895fa WatchSource:0}: Error finding container 026739227bf6dccf28b428749884ab63aa3a86be29e35ba8bf881589104895fa: Status 404 returned error can't find the container with id 026739227bf6dccf28b428749884ab63aa3a86be29e35ba8bf881589104895fa
Oct 30 19:43:56.399402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.399367    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:56.399489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.399435    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:56.399489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:56.399464    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:57.141490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.141456    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" event={"ID":"cba538c7-458e-42de-84b0-54d9e11fdcd7","Type":"ContainerStarted","Data":"bd825c44fb769f6fbe76e51e13938d877492f1bd50ae8e084a8cf3f4f86bdc62"}
Oct 30 19:43:57.141729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.141512    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" event={"ID":"cba538c7-458e-42de-84b0-54d9e11fdcd7","Type":"ContainerStarted","Data":"026739227bf6dccf28b428749884ab63aa3a86be29e35ba8bf881589104895fa"}
Oct 30 19:43:57.143377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.143346    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85" event={"ID":"d94a813f-14ef-4507-9628-2af04dd1d1fa","Type":"ContainerStarted","Data":"6a536e06ed201d7842f2adf7745a545000b870b2423be32b9b40d8466405e12f"}
Oct 30 19:43:57.143513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.143404    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:43:57.143513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.143426    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85" event={"ID":"d94a813f-14ef-4507-9628-2af04dd1d1fa","Type":"ContainerStarted","Data":"cc545b2ae2dca93f9f86eacecf14854ed9ac647830c304c0dbb6123be2ffef64"}
Oct 30 19:43:57.165916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.165873    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" podStartSLOduration=349.165859845 podStartE2EDuration="5m49.165859845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:57.163716631 +0000 UTC m=+2272.066729670" watchObservedRunningTime="2026-10-30 19:43:57.165859845 +0000 UTC m=+2272.068872881"
Oct 30 19:43:57.183487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.183442    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85" podStartSLOduration=12.18343098 podStartE2EDuration="12.18343098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:57.181127434 +0000 UTC m=+2272.084140471" watchObservedRunningTime="2026-10-30 19:43:57.18343098 +0000 UTC m=+2272.086444016"
Oct 30 19:43:57.248328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.248272    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:57.265546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.265513    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89633c90-f3a2-4de0-8b4c-306fabec3394" path="/var/lib/kubelet/pods/89633c90-f3a2-4de0-8b4c-306fabec3394/volumes"
Oct 30 19:43:57.266137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.266115    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc119c21-05da-46e6-a79b-f87ed81e7f95" path="/var/lib/kubelet/pods/bc119c21-05da-46e6-a79b-f87ed81e7f95/volumes"
Oct 30 19:43:57.892087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:57.892055    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:58.029904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:58.029879    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:43:58.060972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:58.060922    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:43:58.062137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:58.062107    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 19:43:58.062333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:58.062306    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 19:43:58.179210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:58.179119    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:58.572140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:58.572051    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:59.020715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.018887    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:43:59.060625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.060594    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 19:43:59.060784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.060640    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 19:43:59.086374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.086296    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73962804-938e-4b9c-b058-a0f071e29de5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73962804-938e-4b9c-b058-a0f071e29de5-plugins-config-volume\") pod \"73962804-938e-4b9c-b058-a0f071e29de5\" (UID: \"73962804-938e-4b9c-b058-a0f071e29de5\") "
Oct 30 19:43:59.086659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.086641    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73962804-938e-4b9c-b058-a0f071e29de5-kube-api-access-kdspd\" (UniqueName: \"kubernetes.io/projected/73962804-938e-4b9c-b058-a0f071e29de5-kube-api-access-kdspd\") pod \"73962804-938e-4b9c-b058-a0f071e29de5\" (UID: \"73962804-938e-4b9c-b058-a0f071e29de5\") "
Oct 30 19:43:59.087097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.087069    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73962804-938e-4b9c-b058-a0f071e29de5-plugins-config-volume" pod "73962804-938e-4b9c-b058-a0f071e29de5" (UID: "73962804-938e-4b9c-b058-a0f071e29de5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:59.087426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.087406    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73962804-938e-4b9c-b058-a0f071e29de5-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.089473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.089445    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73962804-938e-4b9c-b058-a0f071e29de5-kube-api-access-kdspd" pod "73962804-938e-4b9c-b058-a0f071e29de5" (UID: "73962804-938e-4b9c-b058-a0f071e29de5"). InnerVolumeSpecName "kube-api-access-kdspd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:59.145578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.145559    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:43:59.163952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.163923    4925 generic.go:356] "Generic (PLEG): container finished" podID="9abbd9ae-e288-40aa-a2ed-f12cf4073099" containerID="984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53" exitCode=0
Oct 30 19:43:59.164092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.164073    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2" event={"ID":"9abbd9ae-e288-40aa-a2ed-f12cf4073099","Type":"ContainerDied","Data":"984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53"}
Oct 30 19:43:59.164163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.164143    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2" event={"ID":"9abbd9ae-e288-40aa-a2ed-f12cf4073099","Type":"ContainerDied","Data":"5e28802889976b4f80c7bc24c293ef7fbad9a4b66fa2b71e86b482f90fac64b6"}
Oct 30 19:43:59.164302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.164170    4925 scope.go:122] "RemoveContainer" containerID="984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53"
Oct 30 19:43:59.164302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.164270    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"
Oct 30 19:43:59.166839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.166810    4925 generic.go:356] "Generic (PLEG): container finished" podID="73962804-938e-4b9c-b058-a0f071e29de5" containerID="bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e" exitCode=0
Oct 30 19:43:59.167110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.167090    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"
Oct 30 19:43:59.167199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.167162    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt" event={"ID":"73962804-938e-4b9c-b058-a0f071e29de5","Type":"ContainerDied","Data":"bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e"}
Oct 30 19:43:59.167258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.167234    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt" event={"ID":"73962804-938e-4b9c-b058-a0f071e29de5","Type":"ContainerDied","Data":"680e98d07d008d1675dae291a3773427dc90fca742e29a7369183d1f341fbe45"}
Oct 30 19:43:59.179650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.179629    4925 scope.go:122] "RemoveContainer" containerID="984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53"
Oct 30 19:43:59.180070 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:59.180038    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53\": container with ID starting with 984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53 not found: ID does not exist" containerID="984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53"
Oct 30 19:43:59.180152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.180085    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53"} err="failed to get container status \"984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53\": rpc error: code = NotFound desc = could not find container \"984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53\": container with ID starting with 984148cbe6f8f27aae4591aaa38fcf8f0acb8bb83e6ec2f879552cd7f37b1a53 not found: ID does not exist"
Oct 30 19:43:59.180152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.180108    4925 scope.go:122] "RemoveContainer" containerID="bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e"
Oct 30 19:43:59.188991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.188963    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdspd\" (UniqueName: \"kubernetes.io/projected/73962804-938e-4b9c-b058-a0f071e29de5-kube-api-access-kdspd\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.192748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.192728    4925 scope.go:122] "RemoveContainer" containerID="bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e"
Oct 30 19:43:59.219711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.200901    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"]
Oct 30 19:43:59.219711 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:43:59.201026    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e\": container with ID starting with bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e not found: ID does not exist" containerID="bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e"
Oct 30 19:43:59.219711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.201057    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e"} err="failed to get container status \"bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e\": rpc error: code = NotFound desc = could not find container \"bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e\": container with ID starting with bbc109e11696ebc14c60c45ee1d7df033739d35a42e438ea328193c603cae11e not found: ID does not exist"
Oct 30 19:43:59.219711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.209036    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0a6e66-74cdb87c47-dp4zt"]
Oct 30 19:43:59.267567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.267531    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73962804-938e-4b9c-b058-a0f071e29de5" path="/var/lib/kubelet/pods/73962804-938e-4b9c-b058-a0f071e29de5/volumes"
Oct 30 19:43:59.291315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.290553    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abbd9ae-e288-40aa-a2ed-f12cf4073099-kube-api-access-cnnb6\" (UniqueName: \"kubernetes.io/projected/9abbd9ae-e288-40aa-a2ed-f12cf4073099-kube-api-access-cnnb6\") pod \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\" (UID: \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\") "
Oct 30 19:43:59.291315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.290938    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abbd9ae-e288-40aa-a2ed-f12cf4073099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abbd9ae-e288-40aa-a2ed-f12cf4073099-plugins-config-volume\") pod \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\" (UID: \"9abbd9ae-e288-40aa-a2ed-f12cf4073099\") "
Oct 30 19:43:59.312213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.291678    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abbd9ae-e288-40aa-a2ed-f12cf4073099-plugins-config-volume" pod "9abbd9ae-e288-40aa-a2ed-f12cf4073099" (UID: "9abbd9ae-e288-40aa-a2ed-f12cf4073099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:59.312213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.291945    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abbd9ae-e288-40aa-a2ed-f12cf4073099-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.351601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.347461    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:59.365845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.365323    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abbd9ae-e288-40aa-a2ed-f12cf4073099-kube-api-access-cnnb6" pod "9abbd9ae-e288-40aa-a2ed-f12cf4073099" (UID: "9abbd9ae-e288-40aa-a2ed-f12cf4073099"). InnerVolumeSpecName "kube-api-access-cnnb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:59.393583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.393538    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnb6\" (UniqueName: \"kubernetes.io/projected/9abbd9ae-e288-40aa-a2ed-f12cf4073099-kube-api-access-cnnb6\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.515712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.501552    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"]
Oct 30 19:43:59.525860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:43:59.521315    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a6e66-57d9dc94-xl4p2"]
Oct 30 19:44:00.062140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:00.062047    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 19:44:00.062781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:00.062747    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 19:44:00.081710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:00.073394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:00.304718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:00.303481    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:00.869702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:00.869659    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:01.060954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:01.060914    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 19:44:01.061134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:01.060979    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 19:44:01.184378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:01.184337    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:01.265880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:01.265847    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abbd9ae-e288-40aa-a2ed-f12cf4073099" path="/var/lib/kubelet/pods/9abbd9ae-e288-40aa-a2ed-f12cf4073099/volumes"
Oct 30 19:44:01.382034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:01.382001    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:02.062712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:02.062196    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 19:44:02.062712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:02.062273    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 19:44:02.110443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:02.110404    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:02.730526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:02.730490    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:03.066866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:03.066766    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:44:03.067307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:03.067283    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:44:03.068144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:03.068123    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:44:03.463480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:03.463443    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:03.639739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:03.631395    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:03.801768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:03.799622    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:04.213719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:04.213227    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:04.589143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:04.589061    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:04.827993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:04.827957    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:04.995290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:04.995257    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:05.686396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:05.686338    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:06.646666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:06.646618    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:07.808718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:07.801796    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:08.156426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:08.156398    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:44:08.480839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:08.475492    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:09.168951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:09.168914    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:09.992612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:09.992560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:10.659594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:10.659554    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main" containerID="cri-o://a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b" gracePeriod=45
Oct 30 19:44:10.773774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:10.769314    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:10.971249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:10.971224    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:44:11.078551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.078521    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-tmp-dir\") pod \"6951302f-5e56-4c26-ac90-10b26f64724a\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") "
Oct 30 19:44:11.078747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.078678    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-dshm\") pod \"6951302f-5e56-4c26-ac90-10b26f64724a\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") "
Oct 30 19:44:11.078800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.078772    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6951302f-5e56-4c26-ac90-10b26f64724a-kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/6951302f-5e56-4c26-ac90-10b26f64724a-kube-api-access-vwbv2\") pod \"6951302f-5e56-4c26-ac90-10b26f64724a\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") "
Oct 30 19:44:11.078864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.078846    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-tmp-dir" pod "6951302f-5e56-4c26-ac90-10b26f64724a" (UID: "6951302f-5e56-4c26-ac90-10b26f64724a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:11.078927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.078910    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6951302f-5e56-4c26-ac90-10b26f64724a-tls-certs\" (UniqueName: \"kubernetes.io/secret/6951302f-5e56-4c26-ac90-10b26f64724a-tls-certs\") pod \"6951302f-5e56-4c26-ac90-10b26f64724a\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") "
Oct 30 19:44:11.078979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.078965    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-home\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-home\") pod \"6951302f-5e56-4c26-ac90-10b26f64724a\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") "
Oct 30 19:44:11.079197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.079167    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-model-cache\") pod \"6951302f-5e56-4c26-ac90-10b26f64724a\" (UID: \"6951302f-5e56-4c26-ac90-10b26f64724a\") "
Oct 30 19:44:11.079426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.079298    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-home" pod "6951302f-5e56-4c26-ac90-10b26f64724a" (UID: "6951302f-5e56-4c26-ac90-10b26f64724a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:11.079671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.079440    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-model-cache" pod "6951302f-5e56-4c26-ac90-10b26f64724a" (UID: "6951302f-5e56-4c26-ac90-10b26f64724a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:11.079909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.079890    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.079989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.079910    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.079989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.079921    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.081031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.081008    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-dshm" pod "6951302f-5e56-4c26-ac90-10b26f64724a" (UID: "6951302f-5e56-4c26-ac90-10b26f64724a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:11.081188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.081170    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6951302f-5e56-4c26-ac90-10b26f64724a-kube-api-access-vwbv2" pod "6951302f-5e56-4c26-ac90-10b26f64724a" (UID: "6951302f-5e56-4c26-ac90-10b26f64724a"). InnerVolumeSpecName "kube-api-access-vwbv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:11.081250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.081214    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6951302f-5e56-4c26-ac90-10b26f64724a-tls-certs" pod "6951302f-5e56-4c26-ac90-10b26f64724a" (UID: "6951302f-5e56-4c26-ac90-10b26f64724a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:11.181728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.181655    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwbv2\" (UniqueName: \"kubernetes.io/projected/6951302f-5e56-4c26-ac90-10b26f64724a-kube-api-access-vwbv2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.181728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.181724    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6951302f-5e56-4c26-ac90-10b26f64724a-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.181939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.181746    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6951302f-5e56-4c26-ac90-10b26f64724a-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.356174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.356136    4925 generic.go:356] "Generic (PLEG): container finished" podID="6951302f-5e56-4c26-ac90-10b26f64724a" containerID="a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b" exitCode=0
Oct 30 19:44:11.356342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.356186    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"
Oct 30 19:44:11.356342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.356227    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" event={"ID":"6951302f-5e56-4c26-ac90-10b26f64724a","Type":"ContainerDied","Data":"a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b"}
Oct 30 19:44:11.356342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.356272    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s" event={"ID":"6951302f-5e56-4c26-ac90-10b26f64724a","Type":"ContainerDied","Data":"978e6452e9e9d1cefbba7003f2276e0dc67f1d881e8e872662ae2df7f2bb7c7b"}
Oct 30 19:44:11.356342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.356291    4925 scope.go:122] "RemoveContainer" containerID="a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b"
Oct 30 19:44:11.367458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.367432    4925 scope.go:122] "RemoveContainer" containerID="a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b"
Oct 30 19:44:11.367903 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:44:11.367871    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b\": container with ID starting with a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b not found: ID does not exist" containerID="a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b"
Oct 30 19:44:11.367991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.367912    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b"} err="failed to get container status \"a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b\": rpc error: code = NotFound desc = could not find container \"a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b\": container with ID starting with a9e4456b28c0246e404b3c8dee6f7fe9cdbbac1bc536d2dad6f6fbbb442b056b not found: ID does not exist"
Oct 30 19:44:11.377814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.377789    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"]
Oct 30 19:44:11.383010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.382990    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-bce71f/premium-w6-bce71f-kserve-74556658b4-g8p7s"]
Oct 30 19:44:11.687462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:11.686793    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:12.338713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:12.338215    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:12.503367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:12.503327    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:12.914922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:12.914885    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:13.093713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:13.087071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:13.326776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:13.305184    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" path="/var/lib/kubelet/pods/6951302f-5e56-4c26-ac90-10b26f64724a/volumes"
Oct 30 19:44:13.981162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:13.981124    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:14.382425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.382346    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"]
Oct 30 19:44:14.383189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.383169    4925 cpu_manager.go:414] "containerMap: removing container" podUID="73962804-938e-4b9c-b058-a0f071e29de5" containerName="payload-pre-processing"
Oct 30 19:44:14.383270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.383190    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73962804-938e-4b9c-b058-a0f071e29de5" containerName="payload-pre-processing"
Oct 30 19:44:14.383270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.383218    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9abbd9ae-e288-40aa-a2ed-f12cf4073099" containerName="payload-processing"
Oct 30 19:44:14.383270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.383224    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abbd9ae-e288-40aa-a2ed-f12cf4073099" containerName="payload-processing"
Oct 30 19:44:14.383270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.383248    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main"
Oct 30 19:44:14.383270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.383252    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main"
Oct 30 19:44:14.383549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.383374    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abbd9ae-e288-40aa-a2ed-f12cf4073099" containerName="payload-processing"
Oct 30 19:44:14.383549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.383387    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6951302f-5e56-4c26-ac90-10b26f64724a" containerName="main"
Oct 30 19:44:14.383549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.383396    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="73962804-938e-4b9c-b058-a0f071e29de5" containerName="payload-pre-processing"
Oct 30 19:44:14.388884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.388858    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.392894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.392866    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:14.393039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.393022    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:44:14.395481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.395458    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"]
Oct 30 19:44:14.527808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.527778    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda5464d-s56gm\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.527808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.527812    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9dl\" (UniqueName: \"kubernetes.io/projected/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-kube-api-access-hd9dl\") pod \"maas-api-revoke-sub-fda5464d-s56gm\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.528143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.528130    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda5464d-s56gm\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.629406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.629370    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda5464d-s56gm\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.629605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.629470    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda5464d-s56gm\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.629605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.629494    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9dl\" (UniqueName: \"kubernetes.io/projected/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-kube-api-access-hd9dl\") pod \"maas-api-revoke-sub-fda5464d-s56gm\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.630214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.630184    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda5464d-s56gm\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.632155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.632130    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda5464d-s56gm\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.660210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.660188    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9dl\" (UniqueName: \"kubernetes.io/projected/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-kube-api-access-hd9dl\") pod \"maas-api-revoke-sub-fda5464d-s56gm\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.706303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.706278    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:14.782755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:14.782023    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:15.048549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:15.048517    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"]
Oct 30 19:44:15.049532 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:44:15.049508    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7518ab0_ddaf_40ae_9c6e_8647f5b2f82f.slice/crio-3cfe16be9f6e303f7a361da904cf5bfec22e634320ca010ef8d3aafcf64856ca WatchSource:0}: Error finding container 3cfe16be9f6e303f7a361da904cf5bfec22e634320ca010ef8d3aafcf64856ca: Status 404 returned error can't find the container with id 3cfe16be9f6e303f7a361da904cf5bfec22e634320ca010ef8d3aafcf64856ca
Oct 30 19:44:15.504607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:15.504502    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:15.509409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:15.509371    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm" event={"ID":"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f","Type":"ContainerStarted","Data":"de14abf6fd37ffeb5c2c81bc3b9a8dba069874f586dcfbf6925501002cc13b7e"}
Oct 30 19:44:15.510447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:15.509737    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm" event={"ID":"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f","Type":"ContainerStarted","Data":"3cfe16be9f6e303f7a361da904cf5bfec22e634320ca010ef8d3aafcf64856ca"}
Oct 30 19:44:15.529350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:15.529299    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm" podStartSLOduration=1.529283795 podStartE2EDuration="1.529283795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:15.528772299 +0000 UTC m=+2290.431785337" watchObservedRunningTime="2026-10-30 19:44:15.529283795 +0000 UTC m=+2290.432296831"
Oct 30 19:44:16.381420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:16.381316    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:16.519948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:16.519912    4925 generic.go:356] "Generic (PLEG): container finished" podID="e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f" containerID="de14abf6fd37ffeb5c2c81bc3b9a8dba069874f586dcfbf6925501002cc13b7e" exitCode=0
Oct 30 19:44:16.520131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:16.520015    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm" event={"ID":"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f","Type":"ContainerDied","Data":"de14abf6fd37ffeb5c2c81bc3b9a8dba069874f586dcfbf6925501002cc13b7e"}
Oct 30 19:44:17.069013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.065982    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:17.728712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.727574    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:17.797765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.797715    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:17.835018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.834977    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-xks-ca\") pod \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") "
Oct 30 19:44:17.835201 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.835107    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-openshift-ca\") pod \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") "
Oct 30 19:44:17.835481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.835457    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-kube-api-access-hd9dl\" (UniqueName: \"kubernetes.io/projected/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-kube-api-access-hd9dl\") pod \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\" (UID: \"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f\") "
Oct 30 19:44:17.836168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.836130    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-openshift-ca" pod "e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f" (UID: "e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:17.836867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.836838    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:17.838600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.838573    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-kube-api-access-hd9dl" pod "e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f" (UID: "e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f"). InnerVolumeSpecName "kube-api-access-hd9dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:17.839103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.839085    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-xks-ca" pod "e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f" (UID: "e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:17.938797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.938770    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:17.938797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:17.938795    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd9dl\" (UniqueName: \"kubernetes.io/projected/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f-kube-api-access-hd9dl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:18.482450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.480954    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:18.540144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.540112    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm" event={"ID":"e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f","Type":"ContainerDied","Data":"3cfe16be9f6e303f7a361da904cf5bfec22e634320ca010ef8d3aafcf64856ca"}
Oct 30 19:44:18.540330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.540163    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cfe16be9f6e303f7a361da904cf5bfec22e634320ca010ef8d3aafcf64856ca"
Oct 30 19:44:18.540330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.540214    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"
Oct 30 19:44:18.949717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.949675    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"]
Oct 30 19:44:18.950438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.950422    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f" containerName="revoke-keys"
Oct 30 19:44:18.950438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.950438    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f" containerName="revoke-keys"
Oct 30 19:44:18.950550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.950541    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f" containerName="revoke-keys"
Oct 30 19:44:18.954738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.954709    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:18.965739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.963657    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:18.965739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.964290    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:44:18.965739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:18.965626    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"]
Oct 30 19:44:19.064490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.062394    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.064490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.062478    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.064490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.062565    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ks78\" (UniqueName: \"kubernetes.io/projected/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-kube-api-access-8ks78\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.103297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.103200    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:19.189307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.189266    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.189510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.189325    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.189510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.189405    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ks78\" (UniqueName: \"kubernetes.io/projected/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-kube-api-access-8ks78\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.190031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.190009    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.192181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.192156    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.201168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.201109    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ks78\" (UniqueName: \"kubernetes.io/projected/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-kube-api-access-8ks78\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.274844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.274817    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:19.561862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.561830    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"]
Oct 30 19:44:19.562231 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:44:19.562206    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fb6bc5_9583_4c9b_83c8_954b2e38a0b5.slice/crio-b9cc3aabdbe8d370e9125fb76b1fe9ccaea7576010d5b9425b87e20a773bcacf WatchSource:0}: Error finding container b9cc3aabdbe8d370e9125fb76b1fe9ccaea7576010d5b9425b87e20a773bcacf: Status 404 returned error can't find the container with id b9cc3aabdbe8d370e9125fb76b1fe9ccaea7576010d5b9425b87e20a773bcacf
Oct 30 19:44:19.975717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:19.972561    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:20.556613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:20.556578    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq" event={"ID":"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5","Type":"ContainerStarted","Data":"9621c6d3ac88554d4f9401d99a8819935bb8667c4bbeb5cf7af7cfb8fc7f3b2c"}
Oct 30 19:44:20.556806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:20.556628    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq" event={"ID":"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5","Type":"ContainerStarted","Data":"b9cc3aabdbe8d370e9125fb76b1fe9ccaea7576010d5b9425b87e20a773bcacf"}
Oct 30 19:44:20.578961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:20.578912    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq" podStartSLOduration=2.578898643 podStartE2EDuration="2.578898643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:20.572362949 +0000 UTC m=+2295.475375986" watchObservedRunningTime="2026-10-30 19:44:20.578898643 +0000 UTC m=+2295.481911680"
Oct 30 19:44:20.998656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:20.992166    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:21.617713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:21.615536    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:21.617926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:21.617824    4925 generic.go:356] "Generic (PLEG): container finished" podID="77fb6bc5-9583-4c9b-83c8-954b2e38a0b5" containerID="9621c6d3ac88554d4f9401d99a8819935bb8667c4bbeb5cf7af7cfb8fc7f3b2c" exitCode=0
Oct 30 19:44:21.618035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:21.618014    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq" event={"ID":"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5","Type":"ContainerDied","Data":"9621c6d3ac88554d4f9401d99a8819935bb8667c4bbeb5cf7af7cfb8fc7f3b2c"}
Oct 30 19:44:22.487525 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:44:22.487487    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:44:22.769439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:22.769292    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:22.957300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:22.957275    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:23.112326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.112249    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-xks-ca\") pod \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") "
Oct 30 19:44:23.112486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.112357    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-kube-api-access-8ks78\" (UniqueName: \"kubernetes.io/projected/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-kube-api-access-8ks78\") pod \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") "
Oct 30 19:44:23.112486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.112389    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-openshift-ca\") pod \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\" (UID: \"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5\") "
Oct 30 19:44:23.112812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.112778    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-openshift-ca" pod "77fb6bc5-9583-4c9b-83c8-954b2e38a0b5" (UID: "77fb6bc5-9583-4c9b-83c8-954b2e38a0b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:23.113163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.113103    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.114637 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.114614    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-xks-ca" pod "77fb6bc5-9583-4c9b-83c8-954b2e38a0b5" (UID: "77fb6bc5-9583-4c9b-83c8-954b2e38a0b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:23.114867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.114848    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-kube-api-access-8ks78" pod "77fb6bc5-9583-4c9b-83c8-954b2e38a0b5" (UID: "77fb6bc5-9583-4c9b-83c8-954b2e38a0b5"). InnerVolumeSpecName "kube-api-access-8ks78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:23.214675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.214649    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.214675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.214672    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ks78\" (UniqueName: \"kubernetes.io/projected/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5-kube-api-access-8ks78\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.644166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.644135    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"
Oct 30 19:44:23.644557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.644172    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq" event={"ID":"77fb6bc5-9583-4c9b-83c8-954b2e38a0b5","Type":"ContainerDied","Data":"b9cc3aabdbe8d370e9125fb76b1fe9ccaea7576010d5b9425b87e20a773bcacf"}
Oct 30 19:44:23.644557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.644224    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9cc3aabdbe8d370e9125fb76b1fe9ccaea7576010d5b9425b87e20a773bcacf"
Oct 30 19:44:23.915723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:23.915604    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:24.784429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:24.784397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:25.584954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:25.584719    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:26.602223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:26.602190    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:27.322915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:27.322880    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:27.994150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:27.994112    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:28.732284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:28.732246    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:29.069394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:29.069312    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"]
Oct 30 19:44:29.194152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:29.194121    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:29.393709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:29.391190    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:29.887038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:29.886946    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:30.470712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:30.462460    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:30.746261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:30.746170    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:31.354315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:31.354266    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:31.964335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:31.964300    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:33.305482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:33.305438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:33.481335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:33.481300    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:33.659432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:33.659398    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:34.080372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.079666    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr" podUID="22843a64-604b-41bf-b6ea-520f5246159b" containerName="payload-processing" containerID="cri-o://6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d" gracePeriod=25
Oct 30 19:44:34.280775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.278069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:34.462980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.462956    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:44:34.498720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.498656    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"]
Oct 30 19:44:34.499469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.499434    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22843a64-604b-41bf-b6ea-520f5246159b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22843a64-604b-41bf-b6ea-520f5246159b-plugins-config-volume\") pod \"22843a64-604b-41bf-b6ea-520f5246159b\" (UID: \"22843a64-604b-41bf-b6ea-520f5246159b\") "
Oct 30 19:44:34.499797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.499775    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22843a64-604b-41bf-b6ea-520f5246159b-kube-api-access-pc7fn\" (UniqueName: \"kubernetes.io/projected/22843a64-604b-41bf-b6ea-520f5246159b-kube-api-access-pc7fn\") pod \"22843a64-604b-41bf-b6ea-520f5246159b\" (UID: \"22843a64-604b-41bf-b6ea-520f5246159b\") "
Oct 30 19:44:34.500381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.499944    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22843a64-604b-41bf-b6ea-520f5246159b-plugins-config-volume" pod "22843a64-604b-41bf-b6ea-520f5246159b" (UID: "22843a64-604b-41bf-b6ea-520f5246159b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:34.500381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.500079    4925 cpu_manager.go:414] "containerMap: removing container" podUID="22843a64-604b-41bf-b6ea-520f5246159b" containerName="payload-processing"
Oct 30 19:44:34.500381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.500102    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22843a64-604b-41bf-b6ea-520f5246159b" containerName="payload-processing"
Oct 30 19:44:34.500381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.500162    4925 cpu_manager.go:414] "containerMap: removing container" podUID="77fb6bc5-9583-4c9b-83c8-954b2e38a0b5" containerName="revoke-keys"
Oct 30 19:44:34.500381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.500200    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77fb6bc5-9583-4c9b-83c8-954b2e38a0b5" containerName="revoke-keys"
Oct 30 19:44:34.500813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.500445    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="77fb6bc5-9583-4c9b-83c8-954b2e38a0b5" containerName="revoke-keys"
Oct 30 19:44:34.500813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.500475    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="22843a64-604b-41bf-b6ea-520f5246159b" containerName="payload-processing"
Oct 30 19:44:34.500813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.500633    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22843a64-604b-41bf-b6ea-520f5246159b-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.505998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.505950    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22843a64-604b-41bf-b6ea-520f5246159b-kube-api-access-pc7fn" pod "22843a64-604b-41bf-b6ea-520f5246159b" (UID: "22843a64-604b-41bf-b6ea-520f5246159b"). InnerVolumeSpecName "kube-api-access-pc7fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:34.506623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.506601    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.522163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.522136    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"]
Oct 30 19:44:34.602198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.602159    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.602403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.602212    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkxt\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-kube-api-access-jvkxt\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.602476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.602394    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.602581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.602553    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.602638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.602612    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5842785-ef0f-4ddd-ba60-518cae3c7279-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.602739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.602678    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.602785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.602766    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.602841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.602800    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.602957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.602940    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.603124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.603109    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc7fn\" (UniqueName: \"kubernetes.io/projected/22843a64-604b-41bf-b6ea-520f5246159b-kube-api-access-pc7fn\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.705137    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.705349    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.705383    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkxt\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-kube-api-access-jvkxt\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.705484    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.705588    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.705624    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5842785-ef0f-4ddd-ba60-518cae3c7279-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.705653    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.705728    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.705767    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.706348    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.707430    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5842785-ef0f-4ddd-ba60-518cae3c7279-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.708760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.708367    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.717753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.710334    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.717753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.710790    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.717753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.711040    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.717753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.712053    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.729769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.729522    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkxt\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-kube-api-access-jvkxt\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.732659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.731883    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.737384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.736752    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"]
Oct 30 19:44:34.737384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.736978    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh" podUID="86a41b6b-9785-472f-b815-0b6db3d7df76" containerName="maas-api" containerID="cri-o://c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01" gracePeriod=30
Oct 30 19:44:34.767829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.767780    4925 generic.go:356] "Generic (PLEG): container finished" podID="22843a64-604b-41bf-b6ea-520f5246159b" containerID="6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d" exitCode=0
Oct 30 19:44:34.767994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.767970    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr" event={"ID":"22843a64-604b-41bf-b6ea-520f5246159b","Type":"ContainerDied","Data":"6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d"}
Oct 30 19:44:34.768064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.768028    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr" event={"ID":"22843a64-604b-41bf-b6ea-520f5246159b","Type":"ContainerDied","Data":"b38f54f11ba663c25dbb13a759c2ce2fd6de0c257b679c812356ce7b81e1e2ab"}
Oct 30 19:44:34.768064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.768058    4925 scope.go:122] "RemoveContainer" containerID="6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d"
Oct 30 19:44:34.769214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.768226    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"
Oct 30 19:44:34.835295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.835232    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"]
Oct 30 19:44:34.843160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.843131    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-178bcc-58cc896d74-cdxgr"]
Oct 30 19:44:34.849301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.849273    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:34.850348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.850319    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:34.866919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.866894    4925 scope.go:122] "RemoveContainer" containerID="6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d"
Oct 30 19:44:34.867263 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:44:34.867240    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d\": container with ID starting with 6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d not found: ID does not exist" containerID="6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d"
Oct 30 19:44:34.867356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:34.867277    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d"} err="failed to get container status \"6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d\": rpc error: code = NotFound desc = could not find container \"6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d\": container with ID starting with 6e9c0205ab25a29581b378558f0dc50f7efe281ee6940fc116445d6f466dbf0d not found: ID does not exist"
Oct 30 19:44:35.040739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.040604    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:35.058927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.049739    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"]
Oct 30 19:44:35.074851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.074807    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-178bcc-lwfvq"]
Oct 30 19:44:35.411114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.402837    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:35.418794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.418763    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22843a64-604b-41bf-b6ea-520f5246159b" path="/var/lib/kubelet/pods/22843a64-604b-41bf-b6ea-520f5246159b/volumes"
Oct 30 19:44:35.419773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.419751    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77fb6bc5-9583-4c9b-83c8-954b2e38a0b5" path="/var/lib/kubelet/pods/77fb6bc5-9583-4c9b-83c8-954b2e38a0b5/volumes"
Oct 30 19:44:35.517237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.517198    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"]
Oct 30 19:44:35.518349 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:44:35.518319    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5842785_ef0f_4ddd_ba60_518cae3c7279.slice/crio-16265d33a9ca6f62161fe8c88b9712e47ac9f492e986c42e2482f7aad9d313bc WatchSource:0}: Error finding container 16265d33a9ca6f62161fe8c88b9712e47ac9f492e986c42e2482f7aad9d313bc: Status 404 returned error can't find the container with id 16265d33a9ca6f62161fe8c88b9712e47ac9f492e986c42e2482f7aad9d313bc
Oct 30 19:44:35.525772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.522624    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:35.525772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.522774    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:35.525772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.522849    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:35.582835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.579766    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:35.645802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.645772    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:44:35.688230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.688149    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"]
Oct 30 19:44:35.689235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.689214    4925 cpu_manager.go:414] "containerMap: removing container" podUID="86a41b6b-9785-472f-b815-0b6db3d7df76" containerName="maas-api"
Oct 30 19:44:35.689235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.689234    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a41b6b-9785-472f-b815-0b6db3d7df76" containerName="maas-api"
Oct 30 19:44:35.689512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.689391    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a41b6b-9785-472f-b815-0b6db3d7df76" containerName="maas-api"
Oct 30 19:44:35.711723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.706571    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.711723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.711531    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a41b6b-9785-472f-b815-0b6db3d7df76-kube-api-access-9995v\" (UniqueName: \"kubernetes.io/projected/86a41b6b-9785-472f-b815-0b6db3d7df76-kube-api-access-9995v\") pod \"86a41b6b-9785-472f-b815-0b6db3d7df76\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") "
Oct 30 19:44:35.711723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.711671    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a41b6b-9785-472f-b815-0b6db3d7df76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a41b6b-9785-472f-b815-0b6db3d7df76-openshift-service-ca\") pod \"86a41b6b-9785-472f-b815-0b6db3d7df76\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") "
Oct 30 19:44:35.716632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.711770    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-maas-api-tls\") pod \"86a41b6b-9785-472f-b815-0b6db3d7df76\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") "
Oct 30 19:44:35.716632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.711992    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-metrics-certs\") pod \"86a41b6b-9785-472f-b815-0b6db3d7df76\" (UID: \"86a41b6b-9785-472f-b815-0b6db3d7df76\") "
Oct 30 19:44:35.716632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.713790    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a41b6b-9785-472f-b815-0b6db3d7df76-openshift-service-ca" pod "86a41b6b-9785-472f-b815-0b6db3d7df76" (UID: "86a41b6b-9785-472f-b815-0b6db3d7df76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:35.717863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.717835    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-metrics-certs" pod "86a41b6b-9785-472f-b815-0b6db3d7df76" (UID: "86a41b6b-9785-472f-b815-0b6db3d7df76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:35.717992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.717978    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a41b6b-9785-472f-b815-0b6db3d7df76-kube-api-access-9995v" pod "86a41b6b-9785-472f-b815-0b6db3d7df76" (UID: "86a41b6b-9785-472f-b815-0b6db3d7df76"). InnerVolumeSpecName "kube-api-access-9995v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:35.721766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.721737    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"]
Oct 30 19:44:35.723244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.723219    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-maas-api-tls" pod "86a41b6b-9785-472f-b815-0b6db3d7df76" (UID: "86a41b6b-9785-472f-b815-0b6db3d7df76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:35.802184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.802147    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" event={"ID":"e5842785-ef0f-4ddd-ba60-518cae3c7279","Type":"ContainerStarted","Data":"3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d"}
Oct 30 19:44:35.802402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.802214    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" event={"ID":"e5842785-ef0f-4ddd-ba60-518cae3c7279","Type":"ContainerStarted","Data":"16265d33a9ca6f62161fe8c88b9712e47ac9f492e986c42e2482f7aad9d313bc"}
Oct 30 19:44:35.804128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.804100    4925 generic.go:356] "Generic (PLEG): container finished" podID="86a41b6b-9785-472f-b815-0b6db3d7df76" containerID="c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01" exitCode=0
Oct 30 19:44:35.804274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.804142    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"
Oct 30 19:44:35.804274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.804185    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh" event={"ID":"86a41b6b-9785-472f-b815-0b6db3d7df76","Type":"ContainerDied","Data":"c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01"}
Oct 30 19:44:35.804274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.804237    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh" event={"ID":"86a41b6b-9785-472f-b815-0b6db3d7df76","Type":"ContainerDied","Data":"7c3177793ee33ff7438908352fea55008d1663ef943299121eb696e0175216fb"}
Oct 30 19:44:35.804274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.804264    4925 scope.go:122] "RemoveContainer" containerID="c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01"
Oct 30 19:44:35.813869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.813837    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.814002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.813883    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.814002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.813934    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.814134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.814092    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.814197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.814146    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.814197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.814174    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2gkw\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-kube-api-access-v2gkw\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.814322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.814209    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.814322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.814267    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24022281-55e8-4e7f-bbf2-8234ef509e19-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.814322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.814284    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.814920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.814900    4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-metrics-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:35.815030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.815006    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9995v\" (UniqueName: \"kubernetes.io/projected/86a41b6b-9785-472f-b815-0b6db3d7df76-kube-api-access-9995v\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:35.815087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.815040    4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a41b6b-9785-472f-b815-0b6db3d7df76-openshift-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:35.815087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.815055    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a41b6b-9785-472f-b815-0b6db3d7df76-maas-api-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:35.817859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.817844    4925 scope.go:122] "RemoveContainer" containerID="c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01"
Oct 30 19:44:35.818135 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:44:35.818111    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01\": container with ID starting with c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01 not found: ID does not exist" containerID="c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01"
Oct 30 19:44:35.818216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.818144    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01"} err="failed to get container status \"c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01\": rpc error: code = NotFound desc = could not find container \"c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01\": container with ID starting with c92f0932e6f0041b9cdb3ae32c35865f846fc61534dd0a96ceb561bbdd0e4a01 not found: ID does not exist"
Oct 30 19:44:35.827555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.827513    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podStartSLOduration=411.827502031 podStartE2EDuration="6m51.827502031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:35.82550108 +0000 UTC m=+2310.728514117" watchObservedRunningTime="2026-10-30 19:44:35.827502031 +0000 UTC m=+2310.730515067"
Oct 30 19:44:35.848411 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.848310    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"]
Oct 30 19:44:35.850113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.850091    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:35.855964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.855941    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-178bcc-7bc57b696-kndfh"]
Oct 30 19:44:35.917318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917280    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.917486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917398    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24022281-55e8-4e7f-bbf2-8234ef509e19-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.917486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917429    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.917749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917607    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.917749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917637    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.917749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917723    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.917985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917733    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.917985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917848    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.917985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917933    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.918160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.917992    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.918160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.918055    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.918160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.918106    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2gkw\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-kube-api-access-v2gkw\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.918160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.918132    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24022281-55e8-4e7f-bbf2-8234ef509e19-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.918487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.918448    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.920371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.920346    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.920748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.920731    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.933730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.933659    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2gkw\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-kube-api-access-v2gkw\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:35.933730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:35.933658    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:36.052774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.052650    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:36.382134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.382091    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"]
Oct 30 19:44:36.384438 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:44:36.384406    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24022281_55e8_4e7f_bbf2_8234ef509e19.slice/crio-b60b095ece56ec659201bce9877eb46d779c1f589ffbdbd9d0d55af160f16bcd WatchSource:0}: Error finding container b60b095ece56ec659201bce9877eb46d779c1f589ffbdbd9d0d55af160f16bcd: Status 404 returned error can't find the container with id b60b095ece56ec659201bce9877eb46d779c1f589ffbdbd9d0d55af160f16bcd
Oct 30 19:44:36.387653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.387615    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:36.387784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.387706    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:36.387784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.387742    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:36.658709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.657346    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:36.846711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.846320    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" event={"ID":"24022281-55e8-4e7f-bbf2-8234ef509e19","Type":"ContainerStarted","Data":"9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172"}
Oct 30 19:44:36.846711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.846400    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" event={"ID":"24022281-55e8-4e7f-bbf2-8234ef509e19","Type":"ContainerStarted","Data":"b60b095ece56ec659201bce9877eb46d779c1f589ffbdbd9d0d55af160f16bcd"}
Oct 30 19:44:36.851713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.851460    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:44:36.851713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.851512    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:44:36.871594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:36.871525    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podStartSLOduration=412.871508504 podStartE2EDuration="6m52.871508504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:36.870266977 +0000 UTC m=+2311.773280014" watchObservedRunningTime="2026-10-30 19:44:36.871508504 +0000 UTC m=+2311.774521542"
Oct 30 19:44:37.053784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:37.053673    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:37.055548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:37.055516    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.055782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:37.055580    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:44:37.269703 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:37.269666    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a41b6b-9785-472f-b815-0b6db3d7df76" path="/var/lib/kubelet/pods/86a41b6b-9785-472f-b815-0b6db3d7df76/volumes"
Oct 30 19:44:37.787751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:37.779884    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:37.873711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:37.869383    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.873711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:37.869441    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:44:38.054040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:38.053943    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:44:38.054040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:38.053996    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:44:38.800262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:38.800220    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:38.943408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:38.928706    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"]
Oct 30 19:44:38.943408 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:38.934489    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9" podUID="36ac07f1-44d5-427c-9e98-7933ef9d0abd" containerName="maas-api" containerID="cri-o://f7274807097589218639f409ad55741d325704a3335360dcbc791f2dde4f6919" gracePeriod=30
Oct 30 19:44:39.054767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:39.054468    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.054767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:39.054538    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:44:39.100716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:39.088039    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.100716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:39.088100    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:44:39.273448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:39.141928    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:39.483031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:39.482929    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:39.727032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:39.724120    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:39.852772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:39.849707    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.852772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:39.849768    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:44:40.028856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:40.026347    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:40.057743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:40.057703    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.057967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:40.057765    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:44:40.175184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:40.174236    4925 generic.go:356] "Generic (PLEG): container finished" podID="36ac07f1-44d5-427c-9e98-7933ef9d0abd" containerID="f7274807097589218639f409ad55741d325704a3335360dcbc791f2dde4f6919" exitCode=0
Oct 30 19:44:40.175184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:40.174472    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9" event={"ID":"36ac07f1-44d5-427c-9e98-7933ef9d0abd","Type":"ContainerDied","Data":"f7274807097589218639f409ad55741d325704a3335360dcbc791f2dde4f6919"}
Oct 30 19:44:40.283035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:40.283000    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"]
Oct 30 19:44:40.693807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:40.647301    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:40.693807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:40.686310    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"]
Oct 30 19:44:40.749852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:40.744243    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:41.033769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.026469    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdc9\" (UniqueName: \"kubernetes.io/projected/173abd12-7db2-472b-bbe3-7b5d2310f12c-kube-api-access-rvdc9\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s\" (UID: \"173abd12-7db2-472b-bbe3-7b5d2310f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:41.033769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.026548    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173abd12-7db2-472b-bbe3-7b5d2310f12c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s\" (UID: \"173abd12-7db2-472b-bbe3-7b5d2310f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:41.064837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.064797    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.074809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.068750    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:44:41.112825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.066622    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:41.113162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.065486    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.113338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.113311    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:44:41.122597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.122561    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a844e\""
Oct 30 19:44:41.141198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.123811    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a844e-dockercfg-4772r\""
Oct 30 19:44:41.184468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.184424    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdc9\" (UniqueName: \"kubernetes.io/projected/173abd12-7db2-472b-bbe3-7b5d2310f12c-kube-api-access-rvdc9\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s\" (UID: \"173abd12-7db2-472b-bbe3-7b5d2310f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:41.184789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.184769    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173abd12-7db2-472b-bbe3-7b5d2310f12c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s\" (UID: \"173abd12-7db2-472b-bbe3-7b5d2310f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:41.186221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.186194    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173abd12-7db2-472b-bbe3-7b5d2310f12c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s\" (UID: \"173abd12-7db2-472b-bbe3-7b5d2310f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:41.254095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.254062    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdc9\" (UniqueName: \"kubernetes.io/projected/173abd12-7db2-472b-bbe3-7b5d2310f12c-kube-api-access-rvdc9\") pod \"payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s\" (UID: \"173abd12-7db2-472b-bbe3-7b5d2310f12c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:41.332054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.331245    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:41.333318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.333293    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:44:41.362838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.362802    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9" event={"ID":"36ac07f1-44d5-427c-9e98-7933ef9d0abd","Type":"ContainerDied","Data":"a1cefc73b4d72dcf075225a2235161a75b1dd6eac2324ec7b5d3c1f770d10350"}
Oct 30 19:44:41.372452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.372423    4925 scope.go:122] "RemoveContainer" containerID="f7274807097589218639f409ad55741d325704a3335360dcbc791f2dde4f6919"
Oct 30 19:44:41.412759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.395028    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-maas-api-tls\") pod \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") "
Oct 30 19:44:41.412759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.395136    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-metrics-certs\") pod \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") "
Oct 30 19:44:41.412759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.395265    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ac07f1-44d5-427c-9e98-7933ef9d0abd-kube-api-access-l4z44\" (UniqueName: \"kubernetes.io/projected/36ac07f1-44d5-427c-9e98-7933ef9d0abd-kube-api-access-l4z44\") pod \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") "
Oct 30 19:44:41.412759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.395342    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ac07f1-44d5-427c-9e98-7933ef9d0abd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ac07f1-44d5-427c-9e98-7933ef9d0abd-openshift-service-ca\") pod \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\" (UID: \"36ac07f1-44d5-427c-9e98-7933ef9d0abd\") "
Oct 30 19:44:41.416492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.413181    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.428152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.428111    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ac07f1-44d5-427c-9e98-7933ef9d0abd-openshift-service-ca" pod "36ac07f1-44d5-427c-9e98-7933ef9d0abd" (UID: "36ac07f1-44d5-427c-9e98-7933ef9d0abd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:41.489727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.450913    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.519489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.500943    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ac07f1-44d5-427c-9e98-7933ef9d0abd-kube-api-access-l4z44" pod "36ac07f1-44d5-427c-9e98-7933ef9d0abd" (UID: "36ac07f1-44d5-427c-9e98-7933ef9d0abd"). InnerVolumeSpecName "kube-api-access-l4z44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:41.539747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.521746    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4z44\" (UniqueName: \"kubernetes.io/projected/36ac07f1-44d5-427c-9e98-7933ef9d0abd-kube-api-access-l4z44\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:41.539747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.521780    4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ac07f1-44d5-427c-9e98-7933ef9d0abd-openshift-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:41.540137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.539779    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"]
Oct 30 19:44:41.541067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.540988    4925 cpu_manager.go:414] "containerMap: removing container" podUID="36ac07f1-44d5-427c-9e98-7933ef9d0abd" containerName="maas-api"
Oct 30 19:44:41.541272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.541255    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ac07f1-44d5-427c-9e98-7933ef9d0abd" containerName="maas-api"
Oct 30 19:44:41.553742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.548288    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:44:41.553956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.541647    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ac07f1-44d5-427c-9e98-7933ef9d0abd" containerName="maas-api"
Oct 30 19:44:41.635262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.635176    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.663418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.637188    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:41.709719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.709524    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.733029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.720801    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.740932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.737366    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.740932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.737436    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.740932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.737488    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.740932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.737547    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.740932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.737574    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.740932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.737619    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.740932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.737759    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pzc2\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-kube-api-access-5pzc2\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.740932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.737978    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.740932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.738021    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:41.782722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.749727    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.890912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:41.876493    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"]
Oct 30 19:44:42.275437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.265888    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-maas-api-tls" pod "36ac07f1-44d5-427c-9e98-7933ef9d0abd" (UID: "36ac07f1-44d5-427c-9e98-7933ef9d0abd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.278828    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-metrics-certs" pod "36ac07f1-44d5-427c-9e98-7933ef9d0abd" (UID: "36ac07f1-44d5-427c-9e98-7933ef9d0abd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.284858    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.284981    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.285027    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.285077    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.285134    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.285162    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.285205    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.285312    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pzc2\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-kube-api-access-5pzc2\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.285520    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.285610    4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-metrics-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.285630    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ac07f1-44d5-427c-9e98-7933ef9d0abd-maas-api-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.320819    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.321466    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.321739    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.321979    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.323653    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.325190    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.325238    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.325957    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.343219    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.347933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.343272    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:44:42.373715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.369274    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.401759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.388293    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:42.440342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.421944    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:42.440342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.426824    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:44:42.534764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.534447    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.587718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.557368    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:42.604547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.564030    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:42.614384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.614354    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:44:42.642796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.564074    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:42.642796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.576059    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:42.642796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.626949    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:44:42.642796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.604925    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:42.648712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.575459    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:44:42.723712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.698578    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:42.724918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.700915    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:42.725860 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.724020    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 19:44:42.726443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.726419    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:44:42.816759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.805653    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:44:42.817566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.817535    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:44:42.975829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.853725    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"
Oct 30 19:44:42.975829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.854303    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.975829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:42.854353    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:44:43.116485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.116394    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:44:43.133374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.133338    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:44:43.345289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.132898    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"]
Oct 30 19:44:43.345289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.157434    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:43.345289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.274478    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pzc2\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-kube-api-access-5pzc2\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:43.345289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.320802    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:43.428028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.401944    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:43.428028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.403129    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Oct 30 19:44:43.437077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.437043    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:44:43.562714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.517697    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:43.676823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.552348    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:44:43.677171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.677154    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:44:43.736994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.610796    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:44:43.785164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.649447    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:43.821231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.649497    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:43.821231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.659270    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:44:43.821231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.751460    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:43.821231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.649401    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:44:43.821231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.818469    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:43.821231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.819074    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:44:43.836108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.836074    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:44:43.836427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.836412    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:44:43.866284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.864668    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"]
Oct 30 19:44:43.868622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.868243    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 19:44:43.868622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:43.868295    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 19:44:44.143767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.142801    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:44:44.143767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.142864    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:44:44.235311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.234528    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:44:44.342269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.342229    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:44:44.430797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.430493    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:44:44.485852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.445053    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:44.495734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.495676    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:44.512286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.500179    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:44:44.553320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.553277    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:44:44.580317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.580281    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:44:44.583734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.583447    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s" event={"ID":"173abd12-7db2-472b-bbe3-7b5d2310f12c","Type":"ContainerStarted","Data":"d653a7f80ffe4b641cc866e5c8de177dda4be344040f798ff1a9708d537d1ee8"}
Oct 30 19:44:44.615734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.609762    4925 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:44.615734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.610084    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-bce71f-6644698bdb-rc9d9"]
Oct 30 19:44:44.615734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.610179    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:44:44.642387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.642360    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:44:44.668427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.668396    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:44.710534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:44.710504    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:44:45.038466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.038432    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"]
Oct 30 19:44:45.038998 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:44:45.038963    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaffd65_fa65_4d9d_9e74_36dd07b6c891.slice/crio-11332e4ed297154ef6010b341f109fe4f0385d3be19b0085ae2063ba6cb5ebb8 WatchSource:0}: Error finding container 11332e4ed297154ef6010b341f109fe4f0385d3be19b0085ae2063ba6cb5ebb8: Status 404 returned error can't find the container with id 11332e4ed297154ef6010b341f109fe4f0385d3be19b0085ae2063ba6cb5ebb8
Oct 30 19:44:45.043348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.043308    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:45.043441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.043397    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:45.043496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.043467    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:45.053460 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.053437    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Oct 30 19:44:45.053533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.053477    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Oct 30 19:44:45.091598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.091564    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:45.092214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.092124    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:45.092941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.092922    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:44:45.287935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.287856    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ac07f1-44d5-427c-9e98-7933ef9d0abd" path="/var/lib/kubelet/pods/36ac07f1-44d5-427c-9e98-7933ef9d0abd/volumes"
Oct 30 19:44:45.530458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.530416    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:45.629844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.628897    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" event={"ID":"6aaffd65-fa65-4d9d-9e74-36dd07b6c891","Type":"ContainerStarted","Data":"5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760"}
Oct 30 19:44:45.629844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.628978    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" event={"ID":"6aaffd65-fa65-4d9d-9e74-36dd07b6c891","Type":"ContainerStarted","Data":"11332e4ed297154ef6010b341f109fe4f0385d3be19b0085ae2063ba6cb5ebb8"}
Oct 30 19:44:45.641866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.635354    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s" event={"ID":"173abd12-7db2-472b-bbe3-7b5d2310f12c","Type":"ContainerStarted","Data":"f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34"}
Oct 30 19:44:45.641866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.635407    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:45.654270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.654208    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" podStartSLOduration=421.654190732 podStartE2EDuration="7m1.654190732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:45.651304341 +0000 UTC m=+2320.554317382" watchObservedRunningTime="2026-10-30 19:44:45.654190732 +0000 UTC m=+2320.557203770"
Oct 30 19:44:45.674858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:45.674817    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s" podStartSLOduration=5.674802744 podStartE2EDuration="5.674802744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:45.673363874 +0000 UTC m=+2320.576376905" watchObservedRunningTime="2026-10-30 19:44:45.674802744 +0000 UTC m=+2320.577815781"
Oct 30 19:44:46.278071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:46.278041    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:46.278417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:46.278396    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:46.279298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:46.279275    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:44:46.522191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:46.522158    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:46.528259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:46.528177    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:44:46.535956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:46.535912    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 19:44:46.679726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:46.678257    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:47.522745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:47.521822    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:44:47.522745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:47.521914    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 19:44:47.713713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:47.712104    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:47.888945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:47.888804    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:48.069805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.069774    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"]
Oct 30 19:44:48.069983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.069965    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy" containerID="cri-o://e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8" gracePeriod=30
Oct 30 19:44:48.074508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.074480    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"]
Oct 30 19:44:48.074721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.074663    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy" containerID="cri-o://645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b" gracePeriod=30
Oct 30 19:44:48.098028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.097743    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:48.298488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.298414    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:48.514335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.514295    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:48.521316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.521287    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:44:48.521568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.521543    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 19:44:48.826029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.825995    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"]
Oct 30 19:44:48.831833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.831808    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:48.836384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.836357    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:48.836500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.836401    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:48.847995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.847940    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"]
Oct 30 19:44:48.858184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.858049    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:44:48.858184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.858121    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:48.953717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.948942    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:48.970711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.961171    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkk7h\" (UniqueName: \"kubernetes.io/projected/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-kube-api-access-gkk7h\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:48.970711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.961346    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:48.993305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:48.991868    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:49.062827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.062797    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkk7h\" (UniqueName: \"kubernetes.io/projected/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-kube-api-access-gkk7h\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:49.062981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.062872    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:49.063041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.063020    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:49.063703 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.063656    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:49.065575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.065547    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:49.074840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.074815    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkk7h\" (UniqueName: \"kubernetes.io/projected/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-kube-api-access-gkk7h\") pod \"maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:49.158858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.158827    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:49.432713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.432613    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:49.472954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.472924    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"]
Oct 30 19:44:49.473988 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:44:49.473965    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4292a8_327d_4cb3_ab5e_4c3f1a705482.slice/crio-c316eefac18ddb2f44be9d41f084964654ce66ef51e08ce53c5ed12fb91bad97 WatchSource:0}: Error finding container c316eefac18ddb2f44be9d41f084964654ce66ef51e08ce53c5ed12fb91bad97: Status 404 returned error can't find the container with id c316eefac18ddb2f44be9d41f084964654ce66ef51e08ce53c5ed12fb91bad97
Oct 30 19:44:49.520119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.520075    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:44:49.520478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.520144    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 19:44:49.626713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.626401    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:49.888716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:49.887873    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764" event={"ID":"5f4292a8-327d-4cb3-ab5e-4c3f1a705482","Type":"ContainerStarted","Data":"c316eefac18ddb2f44be9d41f084964654ce66ef51e08ce53c5ed12fb91bad97"}
Oct 30 19:44:50.025711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:50.021120    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:50.520275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:50.520223    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:44:50.520635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:50.520565    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 19:44:50.923231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:50.923196    4925 generic.go:356] "Generic (PLEG): container finished" podID="5f4292a8-327d-4cb3-ab5e-4c3f1a705482" containerID="239cb62f5a3c106885e3c1933140c862caa73c882ef841e8e94d21374f079897" exitCode=0
Oct 30 19:44:50.923757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:50.923285    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764" event={"ID":"5f4292a8-327d-4cb3-ab5e-4c3f1a705482","Type":"ContainerDied","Data":"239cb62f5a3c106885e3c1933140c862caa73c882ef841e8e94d21374f079897"}
Oct 30 19:44:51.101713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:51.099273    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:51.520108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:51.520073    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 19:44:51.520314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:51.520149    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 19:44:51.764709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:51.764615    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:52.132849 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:44:52.132812    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:44:52.137075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.137050    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:52.240529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.240449    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-kube-api-access-gkk7h\" (UniqueName: \"kubernetes.io/projected/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-kube-api-access-gkk7h\") pod \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") "
Oct 30 19:44:52.240529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.240501    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-xks-ca\") pod \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") "
Oct 30 19:44:52.240779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.240624    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-openshift-ca\") pod \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\" (UID: \"5f4292a8-327d-4cb3-ab5e-4c3f1a705482\") "
Oct 30 19:44:52.241049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.241019    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-openshift-ca" pod "5f4292a8-327d-4cb3-ab5e-4c3f1a705482" (UID: "5f4292a8-327d-4cb3-ab5e-4c3f1a705482"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:52.241367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.241351    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.243439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.243416    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-kube-api-access-gkk7h" pod "5f4292a8-327d-4cb3-ab5e-4c3f1a705482" (UID: "5f4292a8-327d-4cb3-ab5e-4c3f1a705482"). InnerVolumeSpecName "kube-api-access-gkk7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:52.243520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.243435    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-xks-ca" pod "5f4292a8-327d-4cb3-ab5e-4c3f1a705482" (UID: "5f4292a8-327d-4cb3-ab5e-4c3f1a705482"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:52.343154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.343117    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkk7h\" (UniqueName: \"kubernetes.io/projected/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-kube-api-access-gkk7h\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.343333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.343158    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4292a8-327d-4cb3-ab5e-4c3f1a705482-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.456728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.453304    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:52.523388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.523307    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:52.523826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.523800    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:52.524454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.524434    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:44:52.942275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.942235    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"
Oct 30 19:44:52.942432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.942276    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764" event={"ID":"5f4292a8-327d-4cb3-ab5e-4c3f1a705482","Type":"ContainerDied","Data":"c316eefac18ddb2f44be9d41f084964654ce66ef51e08ce53c5ed12fb91bad97"}
Oct 30 19:44:52.942432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:52.942348    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c316eefac18ddb2f44be9d41f084964654ce66ef51e08ce53c5ed12fb91bad97"
Oct 30 19:44:53.273287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.273249    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:53.475299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.475238    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:44:53.523310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.523281    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"]
Oct 30 19:44:53.524607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.524425    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5f4292a8-327d-4cb3-ab5e-4c3f1a705482" containerName="revoke-keys"
Oct 30 19:44:53.524607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.524448    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4292a8-327d-4cb3-ab5e-4c3f1a705482" containerName="revoke-keys"
Oct 30 19:44:53.524607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.524472    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy"
Oct 30 19:44:53.524607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.524480    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy"
Oct 30 19:44:53.525133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.524726    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4292a8-327d-4cb3-ab5e-4c3f1a705482" containerName="revoke-keys"
Oct 30 19:44:53.525133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.524745    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerName="istio-proxy"
Oct 30 19:44:53.532552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.532164    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.548795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.548770    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"]
Oct 30 19:44:53.556803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.556767    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:44:53.590595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.590518    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-certs\") pod \"e3ee6e4e-8653-4307-a279-f0528a548aef\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") "
Oct 30 19:44:53.590799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.590680    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-credential-socket\") pod \"e3ee6e4e-8653-4307-a279-f0528a548aef\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") "
Oct 30 19:44:53.590799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.590739    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-socket\") pod \"e3ee6e4e-8653-4307-a279-f0528a548aef\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") "
Oct 30 19:44:53.591049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.590801    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-token\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-token\") pod \"e3ee6e4e-8653-4307-a279-f0528a548aef\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") "
Oct 30 19:44:53.591049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.590874    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-kube-api-access-v5dc9\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-kube-api-access-v5dc9\") pod \"e3ee6e4e-8653-4307-a279-f0528a548aef\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") "
Oct 30 19:44:53.591049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.590932    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ee6e4e-8653-4307-a279-f0528a548aef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ee6e4e-8653-4307-a279-f0528a548aef-istiod-ca-cert\") pod \"e3ee6e4e-8653-4307-a279-f0528a548aef\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") "
Oct 30 19:44:53.593608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.592842    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-podinfo\") pod \"e3ee6e4e-8653-4307-a279-f0528a548aef\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") "
Oct 30 19:44:53.593608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.591593    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-socket" pod "e3ee6e4e-8653-4307-a279-f0528a548aef" (UID: "e3ee6e4e-8653-4307-a279-f0528a548aef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.593608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.592975    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-data\") pod \"e3ee6e4e-8653-4307-a279-f0528a548aef\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") "
Oct 30 19:44:53.593608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.593022    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-envoy\") pod \"e3ee6e4e-8653-4307-a279-f0528a548aef\" (UID: \"e3ee6e4e-8653-4307-a279-f0528a548aef\") "
Oct 30 19:44:53.593937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.591827    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-certs" pod "e3ee6e4e-8653-4307-a279-f0528a548aef" (UID: "e3ee6e4e-8653-4307-a279-f0528a548aef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.593937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.592018    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-credential-socket" pod "e3ee6e4e-8653-4307-a279-f0528a548aef" (UID: "e3ee6e4e-8653-4307-a279-f0528a548aef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.593937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.592758    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ee6e4e-8653-4307-a279-f0528a548aef-istiod-ca-cert" pod "e3ee6e4e-8653-4307-a279-f0528a548aef" (UID: "e3ee6e4e-8653-4307-a279-f0528a548aef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:53.594671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.594145    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-data" pod "e3ee6e4e-8653-4307-a279-f0528a548aef" (UID: "e3ee6e4e-8653-4307-a279-f0528a548aef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.595093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.595064    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.595220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.595096    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.595220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.595111    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.595220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.595124    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ee6e4e-8653-4307-a279-f0528a548aef-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.595220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.595138    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.597706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.597663    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-podinfo" pod "e3ee6e4e-8653-4307-a279-f0528a548aef" (UID: "e3ee6e4e-8653-4307-a279-f0528a548aef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:53.597836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.597818    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-envoy" pod "e3ee6e4e-8653-4307-a279-f0528a548aef" (UID: "e3ee6e4e-8653-4307-a279-f0528a548aef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.598515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.598494    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-token" pod "e3ee6e4e-8653-4307-a279-f0528a548aef" (UID: "e3ee6e4e-8653-4307-a279-f0528a548aef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:53.598943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.598923    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-kube-api-access-v5dc9" pod "e3ee6e4e-8653-4307-a279-f0528a548aef" (UID: "e3ee6e4e-8653-4307-a279-f0528a548aef"). InnerVolumeSpecName "kube-api-access-v5dc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:53.605034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.604879    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"]
Oct 30 19:44:53.606075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.606052    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy"
Oct 30 19:44:53.606176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.606077    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy"
Oct 30 19:44:53.606255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.606233    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerName="istio-proxy"
Oct 30 19:44:53.612202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.612180    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.614078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.614058    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"]
Oct 30 19:44:53.617498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.616408    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:53.617498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.616487    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:53.670589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.670561    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7aa217-serving-cert\""
Oct 30 19:44:53.670792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.670657    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:44:53.670792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.670667    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f8msk\""
Oct 30 19:44:53.696157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696127    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-envoy\") pod \"6cfe234f-50c6-4823-abb3-b07e22e285dd\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") "
Oct 30 19:44:53.696316 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696254    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-podinfo\") pod \"6cfe234f-50c6-4823-abb3-b07e22e285dd\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") "
Oct 30 19:44:53.696441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696421    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-socket\") pod \"6cfe234f-50c6-4823-abb3-b07e22e285dd\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") "
Oct 30 19:44:53.696599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696456    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-data\") pod \"6cfe234f-50c6-4823-abb3-b07e22e285dd\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") "
Oct 30 19:44:53.696599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696511    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-kube-api-access-x29qd\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-kube-api-access-x29qd\") pod \"6cfe234f-50c6-4823-abb3-b07e22e285dd\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") "
Oct 30 19:44:53.696599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696544    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-certs\") pod \"6cfe234f-50c6-4823-abb3-b07e22e285dd\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") "
Oct 30 19:44:53.696794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696605    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cfe234f-50c6-4823-abb3-b07e22e285dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfe234f-50c6-4823-abb3-b07e22e285dd-istiod-ca-cert\") pod \"6cfe234f-50c6-4823-abb3-b07e22e285dd\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") "
Oct 30 19:44:53.696794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696732    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-data" pod "6cfe234f-50c6-4823-abb3-b07e22e285dd" (UID: "6cfe234f-50c6-4823-abb3-b07e22e285dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.696794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696753    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-token\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-token\") pod \"6cfe234f-50c6-4823-abb3-b07e22e285dd\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") "
Oct 30 19:44:53.696964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696826    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-credential-socket\") pod \"6cfe234f-50c6-4823-abb3-b07e22e285dd\" (UID: \"6cfe234f-50c6-4823-abb3-b07e22e285dd\") "
Oct 30 19:44:53.696964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.696917    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-certs" pod "6cfe234f-50c6-4823-abb3-b07e22e285dd" (UID: "6cfe234f-50c6-4823-abb3-b07e22e285dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.697214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.697182    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kdxd\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-kube-api-access-2kdxd\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.697340 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.697192    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cfe234f-50c6-4823-abb3-b07e22e285dd-istiod-ca-cert" pod "6cfe234f-50c6-4823-abb3-b07e22e285dd" (UID: "6cfe234f-50c6-4823-abb3-b07e22e285dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:53.697429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.697411    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-credential-socket" pod "6cfe234f-50c6-4823-abb3-b07e22e285dd" (UID: "6cfe234f-50c6-4823-abb3-b07e22e285dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.697605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.697588    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-socket" pod "6cfe234f-50c6-4823-abb3-b07e22e285dd" (UID: "6cfe234f-50c6-4823-abb3-b07e22e285dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.697848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.697787    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.697943    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698021    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698097    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698161    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698329    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698385    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698411    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698508    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5dc9\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-kube-api-access-v5dc9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698524    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698537    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698550    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698564    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfe234f-50c6-4823-abb3-b07e22e285dd-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698580    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698594    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698609    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.698681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.698623    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ee6e4e-8653-4307-a279-f0528a548aef-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.699715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.699185    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-envoy" pod "6cfe234f-50c6-4823-abb3-b07e22e285dd" (UID: "6cfe234f-50c6-4823-abb3-b07e22e285dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.699715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.699319    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-kube-api-access-x29qd" pod "6cfe234f-50c6-4823-abb3-b07e22e285dd" (UID: "6cfe234f-50c6-4823-abb3-b07e22e285dd"). InnerVolumeSpecName "kube-api-access-x29qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:53.699715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.699636    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-podinfo" pod "6cfe234f-50c6-4823-abb3-b07e22e285dd" (UID: "6cfe234f-50c6-4823-abb3-b07e22e285dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:53.700020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.699999    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-token" pod "6cfe234f-50c6-4823-abb3-b07e22e285dd" (UID: "6cfe234f-50c6-4823-abb3-b07e22e285dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:53.800664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.800572    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kdxd\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-kube-api-access-2kdxd\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.800664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.800613    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrhr\" (UniqueName: \"kubernetes.io/projected/e07330c4-415d-437a-bf9f-cb560ce8862c-kube-api-access-6qrhr\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.800947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.800797    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.800947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.800860    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07330c4-415d-437a-bf9f-cb560ce8862c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.800947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.800911    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.800947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.800937    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.801176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801005    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.801176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801085    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.801176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801169    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.801329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801251    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.801329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801270    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-certs\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.801464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801351    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.801464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801394    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.801464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801422    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-metrics-certs\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.801677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801538    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.801677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801545    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.801677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801579    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.801677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801599    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x29qd\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-kube-api-access-x29qd\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.801677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801614    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cfe234f-50c6-4823-abb3-b07e22e285dd-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.801677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801647    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-credential-socket\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.801963 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.801864    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-data\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.802186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.802164    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.803435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.803413    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-envoy\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.804284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.804257    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-podinfo\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.812599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.812572    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kdxd\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-kube-api-access-2kdxd\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.812742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.812663    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-token\") pod \"e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") " pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.871295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.871215    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:53.884736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.876979    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:53.909730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.905068    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-metrics-certs\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.909730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.905165    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrhr\" (UniqueName: \"kubernetes.io/projected/e07330c4-415d-437a-bf9f-cb560ce8862c-kube-api-access-6qrhr\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.909730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.905379    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07330c4-415d-437a-bf9f-cb560ce8862c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.909730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.905542    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.909730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.909243    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.910154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.909945    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07330c4-415d-437a-bf9f-cb560ce8862c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.911456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.911427    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-metrics-certs\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.918390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.918365    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrhr\" (UniqueName: \"kubernetes.io/projected/e07330c4-415d-437a-bf9f-cb560ce8862c-kube-api-access-6qrhr\") pod \"maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.932381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.932354    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:53.960161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.959881    4925 generic.go:356] "Generic (PLEG): container finished" podID="e3ee6e4e-8653-4307-a279-f0528a548aef" containerID="e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8" exitCode=0
Oct 30 19:44:53.960161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.959947    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"
Oct 30 19:44:53.960161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.960003    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" event={"ID":"e3ee6e4e-8653-4307-a279-f0528a548aef","Type":"ContainerDied","Data":"e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8"}
Oct 30 19:44:53.960161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.960041    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b" event={"ID":"e3ee6e4e-8653-4307-a279-f0528a548aef","Type":"ContainerDied","Data":"2b5e60ded537536a45c9169017d81f6f692fdeb9c98086ae4bfa61268bc26805"}
Oct 30 19:44:53.960161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.960062    4925 scope.go:122] "RemoveContainer" containerID="e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8"
Oct 30 19:44:53.962784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.962290    4925 generic.go:356] "Generic (PLEG): container finished" podID="6cfe234f-50c6-4823-abb3-b07e22e285dd" containerID="645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b" exitCode=0
Oct 30 19:44:53.963915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.963510    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"
Oct 30 19:44:53.963915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.963824    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" event={"ID":"6cfe234f-50c6-4823-abb3-b07e22e285dd","Type":"ContainerDied","Data":"645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b"}
Oct 30 19:44:53.963915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.963862    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx" event={"ID":"6cfe234f-50c6-4823-abb3-b07e22e285dd","Type":"ContainerDied","Data":"844946b0c7b9a74b83c71909aa8f2efbdb5bdfedd94b4f2d96865eaca9b5225f"}
Oct 30 19:44:53.980530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.980506    4925 scope.go:122] "RemoveContainer" containerID="e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8"
Oct 30 19:44:53.981205 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:44:53.981179    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8\": container with ID starting with e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8 not found: ID does not exist" containerID="e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8"
Oct 30 19:44:53.981321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.981241    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8"} err="failed to get container status \"e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8\": rpc error: code = NotFound desc = could not find container \"e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8\": container with ID starting with e112f45528519cf8b47a7e6b7d5f3961521306f4f0c3b1864df6a432208eeeb8 not found: ID does not exist"
Oct 30 19:44:53.981321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:53.981267    4925 scope.go:122] "RemoveContainer" containerID="645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b"
Oct 30 19:44:54.027042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.026984    4925 scope.go:122] "RemoveContainer" containerID="645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b"
Oct 30 19:44:54.029988 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:44:54.029906    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b\": container with ID starting with 645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b not found: ID does not exist" containerID="645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b"
Oct 30 19:44:54.029988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.029951    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b"} err="failed to get container status \"645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b\": rpc error: code = NotFound desc = could not find container \"645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b\": container with ID starting with 645b5621630e5303c40995174ea46525803aff5ca4b56434543d21bee9875b3b not found: ID does not exist"
Oct 30 19:44:54.034822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.034793    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"]
Oct 30 19:44:54.041261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.041224    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-hwm7b"]
Oct 30 19:44:54.051730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.051620    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"]
Oct 30 19:44:54.056346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.056320    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bce71f-openshift-default-5d99f47777-drsfx"]
Oct 30 19:44:54.215831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.215237    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"]
Oct 30 19:44:54.219188 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:44:54.219129    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7ef05f_31eb_44f2_a29e_ebc0673a13e3.slice/crio-6943b8269efb9dac6ce254f8aa1298fab6282af95dca0c8d4477d1b5f273cea3 WatchSource:0}: Error finding container 6943b8269efb9dac6ce254f8aa1298fab6282af95dca0c8d4477d1b5f273cea3: Status 404 returned error can't find the container with id 6943b8269efb9dac6ce254f8aa1298fab6282af95dca0c8d4477d1b5f273cea3
Oct 30 19:44:54.223105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.223047    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:54.223336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.223303    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:54.223506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.223368    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:54.265929 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:44:54.265886    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07330c4_415d_437a_bf9f_cb560ce8862c.slice/crio-cb6da0d91cf2ef5e7fa94c1808072d710b011fc10501092d4902a8ea31e3e8a3 WatchSource:0}: Error finding container cb6da0d91cf2ef5e7fa94c1808072d710b011fc10501092d4902a8ea31e3e8a3: Status 404 returned error can't find the container with id cb6da0d91cf2ef5e7fa94c1808072d710b011fc10501092d4902a8ea31e3e8a3
Oct 30 19:44:54.266322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.266298    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"]
Oct 30 19:44:54.559790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.553836    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:54.973561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.973520    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" event={"ID":"db7ef05f-31eb-44f2-a29e-ebc0673a13e3","Type":"ContainerStarted","Data":"e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27"}
Oct 30 19:44:54.973789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.973626    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" event={"ID":"db7ef05f-31eb-44f2-a29e-ebc0673a13e3","Type":"ContainerStarted","Data":"6943b8269efb9dac6ce254f8aa1298fab6282af95dca0c8d4477d1b5f273cea3"}
Oct 30 19:44:54.975564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.975535    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5" event={"ID":"e07330c4-415d-437a-bf9f-cb560ce8862c","Type":"ContainerStarted","Data":"ceb715b164a82c16bf0c625e26b33bbc88de94f3a942ee18648b9999401ae30b"}
Oct 30 19:44:54.975800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.975766    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5" event={"ID":"e07330c4-415d-437a-bf9f-cb560ce8862c","Type":"ContainerStarted","Data":"cb6da0d91cf2ef5e7fa94c1808072d710b011fc10501092d4902a8ea31e3e8a3"}
Oct 30 19:44:54.975800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:54.975799    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:44:55.000112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.000057    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" podStartSLOduration=431.000040918 podStartE2EDuration="7m11.000040918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:54.997125674 +0000 UTC m=+2329.900138712" watchObservedRunningTime="2026-10-30 19:44:55.000040918 +0000 UTC m=+2329.903053980"
Oct 30 19:44:55.030290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.030229    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5" podStartSLOduration=51.657561169 podStartE2EDuration="52.030212873s" totalImagesPullingTime="372.651704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:55.023523428 +0000 UTC m=+2329.926536464" watchObservedRunningTime="2026-10-30 19:44:55.030212873 +0000 UTC m=+2329.933225910"
Oct 30 19:44:55.279395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.279301    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cfe234f-50c6-4823-abb3-b07e22e285dd" path="/var/lib/kubelet/pods/6cfe234f-50c6-4823-abb3-b07e22e285dd/volumes"
Oct 30 19:44:55.279977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.279958    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ee6e4e-8653-4307-a279-f0528a548aef" path="/var/lib/kubelet/pods/e3ee6e4e-8653-4307-a279-f0528a548aef/volumes"
Oct 30 19:44:55.440738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.440386    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:55.830635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.830600    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"]
Oct 30 19:44:55.831078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.830921    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy" containerID="cri-o://49da8472459ae0646c130ad78f38609571c853c61bfb62c60a34b1242977fe47" gracePeriod=30
Oct 30 19:44:55.873834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.873783    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:44:55.874677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.874642    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:55.875211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:55.874762    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 19:44:56.086888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:56.086796    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:56.693151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:56.693121    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:44:56.903564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:56.903525    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:56.907516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:56.907153    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:56.907516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:56.907212    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 19:44:57.774716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:57.766673    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:57.873414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:57.873383    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:57.873592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:57.873445    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 19:44:58.251204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:58.251167    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:58.419933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:58.418600    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:58.873228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:58.873189    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:58.873417 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:58.873279    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 19:44:58.977908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:58.977867    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:59.145510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:59.145262    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:59.313725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:59.310627    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:59.873024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:59.872995    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:44:59.873187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:59.873053    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 19:44:59.984733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:44:59.978409    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:00.139821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.139796    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"]
Oct 30 19:45:00.145590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.145568    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:00.153724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.153670    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"]
Oct 30 19:45:00.158016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.157989    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:45:00.178128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.178084    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wq8\" (UniqueName: \"kubernetes.io/projected/32e94e10-b901-4c30-8080-42b33ce1a0be-kube-api-access-n6wq8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z\" (UID: \"32e94e10-b901-4c30-8080-42b33ce1a0be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:00.178386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.178361    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e94e10-b901-4c30-8080-42b33ce1a0be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z\" (UID: \"32e94e10-b901-4c30-8080-42b33ce1a0be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:00.280856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.280818    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e94e10-b901-4c30-8080-42b33ce1a0be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z\" (UID: \"32e94e10-b901-4c30-8080-42b33ce1a0be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:00.281028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.281004    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wq8\" (UniqueName: \"kubernetes.io/projected/32e94e10-b901-4c30-8080-42b33ce1a0be-kube-api-access-n6wq8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z\" (UID: \"32e94e10-b901-4c30-8080-42b33ce1a0be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:00.281636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.281611    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e94e10-b901-4c30-8080-42b33ce1a0be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z\" (UID: \"32e94e10-b901-4c30-8080-42b33ce1a0be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:00.293378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.293346    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wq8\" (UniqueName: \"kubernetes.io/projected/32e94e10-b901-4c30-8080-42b33ce1a0be-kube-api-access-n6wq8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z\" (UID: \"32e94e10-b901-4c30-8080-42b33ce1a0be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:00.472530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.472440    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:00.742787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.733997    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:00.821178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.821142    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"]
Oct 30 19:45:00.827029 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:00.826994    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e94e10_b901_4c30_8080_42b33ce1a0be.slice/crio-d25621c8d44e56b3eb383eaa6da5c3ff288ae0ce405649bf64c78dcbade03929 WatchSource:0}: Error finding container d25621c8d44e56b3eb383eaa6da5c3ff288ae0ce405649bf64c78dcbade03929: Status 404 returned error can't find the container with id d25621c8d44e56b3eb383eaa6da5c3ff288ae0ce405649bf64c78dcbade03929
Oct 30 19:45:00.874218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.874194    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 19:45:00.874346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.874264    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 19:45:00.993399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:00.993327    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:45:01.106240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.106205    4925 generic.go:356] "Generic (PLEG): container finished" podID="51928b0b-54d8-4959-b115-7f091a8c8700" containerID="49da8472459ae0646c130ad78f38609571c853c61bfb62c60a34b1242977fe47" exitCode=0
Oct 30 19:45:01.106447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.106285    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" event={"ID":"51928b0b-54d8-4959-b115-7f091a8c8700","Type":"ContainerDied","Data":"49da8472459ae0646c130ad78f38609571c853c61bfb62c60a34b1242977fe47"}
Oct 30 19:45:01.107994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.107965    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" event={"ID":"32e94e10-b901-4c30-8080-42b33ce1a0be","Type":"ContainerStarted","Data":"d25621c8d44e56b3eb383eaa6da5c3ff288ae0ce405649bf64c78dcbade03929"}
Oct 30 19:45:01.177750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.177725    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:45:01.223805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.223770    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"]
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.224750    4925 cpu_manager.go:414] "containerMap: removing container" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy"
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.224773    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy"
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.225057    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" containerName="istio-proxy"
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.225554    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-socket\") pod \"51928b0b-54d8-4959-b115-7f091a8c8700\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") "
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.225715    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-envoy\") pod \"51928b0b-54d8-4959-b115-7f091a8c8700\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") "
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.225905    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-certs\") pod \"51928b0b-54d8-4959-b115-7f091a8c8700\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") "
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.225949    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-data\") pod \"51928b0b-54d8-4959-b115-7f091a8c8700\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") "
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.226024    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51928b0b-54d8-4959-b115-7f091a8c8700-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51928b0b-54d8-4959-b115-7f091a8c8700-istio-podinfo\") pod \"51928b0b-54d8-4959-b115-7f091a8c8700\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") "
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.226067    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51928b0b-54d8-4959-b115-7f091a8c8700-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51928b0b-54d8-4959-b115-7f091a8c8700-istiod-ca-cert\") pod \"51928b0b-54d8-4959-b115-7f091a8c8700\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") "
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.226121    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-kube-api-access-hd6h2\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-kube-api-access-hd6h2\") pod \"51928b0b-54d8-4959-b115-7f091a8c8700\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") "
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.226180    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-istio-token\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-istio-token\") pod \"51928b0b-54d8-4959-b115-7f091a8c8700\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") "
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.226226    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-credential-socket\") pod \"51928b0b-54d8-4959-b115-7f091a8c8700\" (UID: \"51928b0b-54d8-4959-b115-7f091a8c8700\") "
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.227291    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-credential-socket" pod "51928b0b-54d8-4959-b115-7f091a8c8700" (UID: "51928b0b-54d8-4959-b115-7f091a8c8700"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.227599    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-socket" pod "51928b0b-54d8-4959-b115-7f091a8c8700" (UID: "51928b0b-54d8-4959-b115-7f091a8c8700"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.228887    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-data" pod "51928b0b-54d8-4959-b115-7f091a8c8700" (UID: "51928b0b-54d8-4959-b115-7f091a8c8700"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.229200    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-certs" pod "51928b0b-54d8-4959-b115-7f091a8c8700" (UID: "51928b0b-54d8-4959-b115-7f091a8c8700"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:01.229478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.229385    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51928b0b-54d8-4959-b115-7f091a8c8700-istiod-ca-cert" pod "51928b0b-54d8-4959-b115-7f091a8c8700" (UID: "51928b0b-54d8-4959-b115-7f091a8c8700"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:01.242821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.241428    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-kube-api-access-hd6h2" pod "51928b0b-54d8-4959-b115-7f091a8c8700" (UID: "51928b0b-54d8-4959-b115-7f091a8c8700"). InnerVolumeSpecName "kube-api-access-hd6h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:01.248887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.248118    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"]
Oct 30 19:45:01.268766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.249469    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:01.268766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.260047    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:01.273261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.273221    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-envoy" pod "51928b0b-54d8-4959-b115-7f091a8c8700" (UID: "51928b0b-54d8-4959-b115-7f091a8c8700"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:01.275977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.275449    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-istio-token" pod "51928b0b-54d8-4959-b115-7f091a8c8700" (UID: "51928b0b-54d8-4959-b115-7f091a8c8700"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:01.288726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.286867    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51928b0b-54d8-4959-b115-7f091a8c8700-istio-podinfo" pod "51928b0b-54d8-4959-b115-7f091a8c8700" (UID: "51928b0b-54d8-4959-b115-7f091a8c8700"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:01.300091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.300015    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"]
Oct 30 19:45:01.307212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.307185    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"]
Oct 30 19:45:01.338367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.312109    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:01.338663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.337659    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75816276-0209-464f-8a2a-bd0faca897e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc\" (UID: \"75816276-0209-464f-8a2a-bd0faca897e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:01.338843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.338818    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpz5r\" (UniqueName: \"kubernetes.io/projected/75816276-0209-464f-8a2a-bd0faca897e5-kube-api-access-vpz5r\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc\" (UID: \"75816276-0209-464f-8a2a-bd0faca897e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:01.338927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.338874    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvn7v\" (UniqueName: \"kubernetes.io/projected/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-kube-api-access-tvn7v\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q\" (UID: \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:01.339011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.338932    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q\" (UID: \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:01.339294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.339271    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:01.339294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.339294    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:01.339470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.339309    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51928b0b-54d8-4959-b115-7f091a8c8700-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:01.339470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.339324    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51928b0b-54d8-4959-b115-7f091a8c8700-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:01.339470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.339341    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd6h2\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-kube-api-access-hd6h2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:01.339470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.339356    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51928b0b-54d8-4959-b115-7f091a8c8700-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:01.339470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.339370    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:01.339470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.339388    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:01.339470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.339405    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51928b0b-54d8-4959-b115-7f091a8c8700-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:01.351903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.351762    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"]
Oct 30 19:45:01.352134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.351880    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:01.359332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.359291    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"]
Oct 30 19:45:01.359738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.359716    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:01.367000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.364045    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"]
Oct 30 19:45:01.367000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.365465    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:01.369177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.369142    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"]
Oct 30 19:45:01.369420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.369404    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:01.377599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.377474    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"]
Oct 30 19:45:01.379924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.379741    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:01.391375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.391350    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"]
Oct 30 19:45:01.393653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.393497    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:01.400979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.400137    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"]
Oct 30 19:45:01.400979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.400219    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"]
Oct 30 19:45:01.400979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.400247    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"]
Oct 30 19:45:01.400979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.400355    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:01.414458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.414438    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"]
Oct 30 19:45:01.425282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.424839    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"]
Oct 30 19:45:01.430304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.430261    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"]
Oct 30 19:45:01.445866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.445841    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv\" (UID: \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:01.446434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.446413    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbk72\" (UniqueName: \"kubernetes.io/projected/d1c30400-2461-45d1-a898-8ad75d20331f-kube-api-access-vbk72\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4\" (UID: \"d1c30400-2461-45d1-a898-8ad75d20331f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:01.446616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.446602    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx\" (UID: \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:01.446755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.446742    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6b4n\" (UniqueName: \"kubernetes.io/projected/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-kube-api-access-k6b4n\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx\" (UID: \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:01.447031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.447015    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnss7\" (UniqueName: \"kubernetes.io/projected/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-kube-api-access-jnss7\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh\" (UID: \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:01.448833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.448810    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh\" (UID: \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:01.448983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.448968    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrs9\" (UniqueName: \"kubernetes.io/projected/9455f850-77c8-474c-b383-cecddf457d92-kube-api-access-xlrs9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f\" (UID: \"9455f850-77c8-474c-b383-cecddf457d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:01.449164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.449150    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fb7c2b-8b55-4f61-b29c-1f2093448e31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v\" (UID: \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:01.449285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.449270    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c30400-2461-45d1-a898-8ad75d20331f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4\" (UID: \"d1c30400-2461-45d1-a898-8ad75d20331f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:01.449507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.449491    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktg84\" (UniqueName: \"kubernetes.io/projected/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-kube-api-access-ktg84\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv\" (UID: \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:01.449636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.449622    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75816276-0209-464f-8a2a-bd0faca897e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc\" (UID: \"75816276-0209-464f-8a2a-bd0faca897e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:01.449840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.449820    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqljj\" (UniqueName: \"kubernetes.io/projected/61fb7c2b-8b55-4f61-b29c-1f2093448e31-kube-api-access-kqljj\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v\" (UID: \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:01.450126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.449967    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpz5r\" (UniqueName: \"kubernetes.io/projected/75816276-0209-464f-8a2a-bd0faca897e5-kube-api-access-vpz5r\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc\" (UID: \"75816276-0209-464f-8a2a-bd0faca897e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:01.450255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.450228    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d051361-c3c7-4b15-8fb6-1462d7433649-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs\" (UID: \"2d051361-c3c7-4b15-8fb6-1462d7433649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:01.450381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.450366    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvn7v\" (UniqueName: \"kubernetes.io/projected/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-kube-api-access-tvn7v\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q\" (UID: \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:01.450488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.450476    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9455f850-77c8-474c-b383-cecddf457d92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f\" (UID: \"9455f850-77c8-474c-b383-cecddf457d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:01.450618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.450605    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q\" (UID: \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:01.450739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.450721    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xq2\" (UniqueName: \"kubernetes.io/projected/2d051361-c3c7-4b15-8fb6-1462d7433649-kube-api-access-x4xq2\") pod \"maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs\" (UID: \"2d051361-c3c7-4b15-8fb6-1462d7433649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:01.453730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.453710    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75816276-0209-464f-8a2a-bd0faca897e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc\" (UID: \"75816276-0209-464f-8a2a-bd0faca897e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:01.455486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.455294    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q\" (UID: \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:01.469430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.469399    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpz5r\" (UniqueName: \"kubernetes.io/projected/75816276-0209-464f-8a2a-bd0faca897e5-kube-api-access-vpz5r\") pod \"maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc\" (UID: \"75816276-0209-464f-8a2a-bd0faca897e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:01.473765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.472806    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvn7v\" (UniqueName: \"kubernetes.io/projected/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-kube-api-access-tvn7v\") pod \"maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q\" (UID: \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:01.481817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.481793    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"]
Oct 30 19:45:01.541457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.541377    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:01.553941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.553909    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx\" (UID: \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:01.554095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.553956    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfsg\" (UniqueName: \"kubernetes.io/projected/55dc9932-b47f-4181-b57f-1f021ffb67b9-kube-api-access-kvfsg\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn\" (UID: \"55dc9932-b47f-4181-b57f-1f021ffb67b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:01.554095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.554012    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6b4n\" (UniqueName: \"kubernetes.io/projected/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-kube-api-access-k6b4n\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx\" (UID: \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:01.554222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.554096    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnss7\" (UniqueName: \"kubernetes.io/projected/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-kube-api-access-jnss7\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh\" (UID: \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:01.554222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.554142    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh\" (UID: \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:01.554222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.554173    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrs9\" (UniqueName: \"kubernetes.io/projected/9455f850-77c8-474c-b383-cecddf457d92-kube-api-access-xlrs9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f\" (UID: \"9455f850-77c8-474c-b383-cecddf457d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:01.555236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.555212    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh\" (UID: \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:01.555382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.555248    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx\" (UID: \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:01.555486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.555380    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fb7c2b-8b55-4f61-b29c-1f2093448e31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v\" (UID: \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:01.555998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.555975    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c30400-2461-45d1-a898-8ad75d20331f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4\" (UID: \"d1c30400-2461-45d1-a898-8ad75d20331f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:01.556276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.556255    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktg84\" (UniqueName: \"kubernetes.io/projected/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-kube-api-access-ktg84\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv\" (UID: \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:01.556488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.556469    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqljj\" (UniqueName: \"kubernetes.io/projected/61fb7c2b-8b55-4f61-b29c-1f2093448e31-kube-api-access-kqljj\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v\" (UID: \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:01.556638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.556621    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d051361-c3c7-4b15-8fb6-1462d7433649-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs\" (UID: \"2d051361-c3c7-4b15-8fb6-1462d7433649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:01.556865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.556845    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9455f850-77c8-474c-b383-cecddf457d92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f\" (UID: \"9455f850-77c8-474c-b383-cecddf457d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:01.556992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.556972    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c30400-2461-45d1-a898-8ad75d20331f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4\" (UID: \"d1c30400-2461-45d1-a898-8ad75d20331f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:01.557069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.556295    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fb7c2b-8b55-4f61-b29c-1f2093448e31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v\" (UID: \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:01.557165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.557148    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xq2\" (UniqueName: \"kubernetes.io/projected/2d051361-c3c7-4b15-8fb6-1462d7433649-kube-api-access-x4xq2\") pod \"maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs\" (UID: \"2d051361-c3c7-4b15-8fb6-1462d7433649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:01.557313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.557295    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55dc9932-b47f-4181-b57f-1f021ffb67b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn\" (UID: \"55dc9932-b47f-4181-b57f-1f021ffb67b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:01.557443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.557427    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv\" (UID: \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:01.557564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.557548    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbk72\" (UniqueName: \"kubernetes.io/projected/d1c30400-2461-45d1-a898-8ad75d20331f-kube-api-access-vbk72\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4\" (UID: \"d1c30400-2461-45d1-a898-8ad75d20331f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:01.558341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.558316    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9455f850-77c8-474c-b383-cecddf457d92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f\" (UID: \"9455f850-77c8-474c-b383-cecddf457d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:01.558783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.558762    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d051361-c3c7-4b15-8fb6-1462d7433649-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs\" (UID: \"2d051361-c3c7-4b15-8fb6-1462d7433649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:01.559287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.559026    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv\" (UID: \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:01.565824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.565797    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6b4n\" (UniqueName: \"kubernetes.io/projected/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-kube-api-access-k6b4n\") pod \"maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx\" (UID: \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:01.566560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.566540    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrs9\" (UniqueName: \"kubernetes.io/projected/9455f850-77c8-474c-b383-cecddf457d92-kube-api-access-xlrs9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f\" (UID: \"9455f850-77c8-474c-b383-cecddf457d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:01.567859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.567833    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"]
Oct 30 19:45:01.568289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.568273    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqljj\" (UniqueName: \"kubernetes.io/projected/61fb7c2b-8b55-4f61-b29c-1f2093448e31-kube-api-access-kqljj\") pod \"maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v\" (UID: \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:01.572527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.572505    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"]
Oct 30 19:45:01.574054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.574027    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktg84\" (UniqueName: \"kubernetes.io/projected/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-kube-api-access-ktg84\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv\" (UID: \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:01.576179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.576156    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xq2\" (UniqueName: \"kubernetes.io/projected/2d051361-c3c7-4b15-8fb6-1462d7433649-kube-api-access-x4xq2\") pod \"maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs\" (UID: \"2d051361-c3c7-4b15-8fb6-1462d7433649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:01.576844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.576822    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnss7\" (UniqueName: \"kubernetes.io/projected/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-kube-api-access-jnss7\") pod \"maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh\" (UID: \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:01.577053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.577040    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:01.577470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.577448    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbk72\" (UniqueName: \"kubernetes.io/projected/d1c30400-2461-45d1-a898-8ad75d20331f-kube-api-access-vbk72\") pod \"maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4\" (UID: \"d1c30400-2461-45d1-a898-8ad75d20331f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:01.577617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.577601    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"]
Oct 30 19:45:01.602821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.602781    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:01.661305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.661273    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55dc9932-b47f-4181-b57f-1f021ffb67b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn\" (UID: \"55dc9932-b47f-4181-b57f-1f021ffb67b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:01.661442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.661348    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfsg\" (UniqueName: \"kubernetes.io/projected/55dc9932-b47f-4181-b57f-1f021ffb67b9-kube-api-access-kvfsg\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn\" (UID: \"55dc9932-b47f-4181-b57f-1f021ffb67b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:01.662971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.662944    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55dc9932-b47f-4181-b57f-1f021ffb67b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn\" (UID: \"55dc9932-b47f-4181-b57f-1f021ffb67b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:01.680535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.680284    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:01.681236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.681206    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfsg\" (UniqueName: \"kubernetes.io/projected/55dc9932-b47f-4181-b57f-1f021ffb67b9-kube-api-access-kvfsg\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn\" (UID: \"55dc9932-b47f-4181-b57f-1f021ffb67b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:01.694112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.694082    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:01.707239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.707215    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:01.722492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.722192    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:01.740603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.740576    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:01.759350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.759145    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:01.776798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.776659    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:01.857145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.856884    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:01.901219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.894275    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"]
Oct 30 19:45:01.901219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.894658    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:45:01.901219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.896904    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:45:01.901219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.898259    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:45:01.905485 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:01.905429    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75816276_0209_464f_8a2a_bd0faca897e5.slice/crio-e8db850ccde7bc74f02cbbf8240e4f62f4bd60b6ad5f5e09a8e8209ac1d7175a WatchSource:0}: Error finding container e8db850ccde7bc74f02cbbf8240e4f62f4bd60b6ad5f5e09a8e8209ac1d7175a: Status 404 returned error can't find the container with id e8db850ccde7bc74f02cbbf8240e4f62f4bd60b6ad5f5e09a8e8209ac1d7175a
Oct 30 19:45:01.925369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:01.922014    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"]
Oct 30 19:45:02.004992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.004563    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"]
Oct 30 19:45:02.029066 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:02.029001    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d051361_c3c7_4b15_8fb6_1462d7433649.slice/crio-e9ea8439934e3c1230093dd7dff0d2d74e740f3b01c21daf4f3420bb733d74fd WatchSource:0}: Error finding container e9ea8439934e3c1230093dd7dff0d2d74e740f3b01c21daf4f3420bb733d74fd: Status 404 returned error can't find the container with id e9ea8439934e3c1230093dd7dff0d2d74e740f3b01c21daf4f3420bb733d74fd
Oct 30 19:45:02.123197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.123150    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn" event={"ID":"51928b0b-54d8-4959-b115-7f091a8c8700","Type":"ContainerDied","Data":"50fcf05bed931432b2b18ffd8715d63e55255f2c7b854c2501738533f4d12676"}
Oct 30 19:45:02.123407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.123237    4925 scope.go:122] "RemoveContainer" containerID="49da8472459ae0646c130ad78f38609571c853c61bfb62c60a34b1242977fe47"
Oct 30 19:45:02.123491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.123457    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"
Oct 30 19:45:02.128125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.128093    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" event={"ID":"32e94e10-b901-4c30-8080-42b33ce1a0be","Type":"ContainerStarted","Data":"a4eab4bb07534e1e2ac88f0f32ad6d3574e5e175a3e452818ae90e726b7944e0"}
Oct 30 19:45:02.131005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.130977    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" event={"ID":"2d051361-c3c7-4b15-8fb6-1462d7433649","Type":"ContainerStarted","Data":"e9ea8439934e3c1230093dd7dff0d2d74e740f3b01c21daf4f3420bb733d74fd"}
Oct 30 19:45:02.132540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.132505    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" event={"ID":"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de","Type":"ContainerStarted","Data":"3cf66983da4ed1de392751b8cec081643a8cc4bae6a606f13d5bda8c2c4e6488"}
Oct 30 19:45:02.133819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.133795    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" event={"ID":"75816276-0209-464f-8a2a-bd0faca897e5","Type":"ContainerStarted","Data":"e8db850ccde7bc74f02cbbf8240e4f62f4bd60b6ad5f5e09a8e8209ac1d7175a"}
Oct 30 19:45:02.152102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.152052    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" podStartSLOduration=2.152035747 podStartE2EDuration="2.152035747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.146898658 +0000 UTC m=+2337.049911697" watchObservedRunningTime="2026-10-30 19:45:02.152035747 +0000 UTC m=+2337.055048783"
Oct 30 19:45:02.186523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.186484    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"]
Oct 30 19:45:02.192454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.192422    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-v9tpn"]
Oct 30 19:45:02.238144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.236778    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"]
Oct 30 19:45:02.256918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.256885    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"]
Oct 30 19:45:02.263601 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:02.263558    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9455f850_77c8_474c_b383_cecddf457d92.slice/crio-9c7d9921b3ec3574e6dd0187a03cbaa2a4103b5cb1cdd28d517513aaa4a715f8 WatchSource:0}: Error finding container 9c7d9921b3ec3574e6dd0187a03cbaa2a4103b5cb1cdd28d517513aaa4a715f8: Status 404 returned error can't find the container with id 9c7d9921b3ec3574e6dd0187a03cbaa2a4103b5cb1cdd28d517513aaa4a715f8
Oct 30 19:45:02.264129 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:02.264104    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fb7c2b_8b55_4f61_b29c_1f2093448e31.slice/crio-8b179ef6589eaf9b63b30caae7a4af4a15433b5ea8778a0c967bfe3b6cc91940 WatchSource:0}: Error finding container 8b179ef6589eaf9b63b30caae7a4af4a15433b5ea8778a0c967bfe3b6cc91940: Status 404 returned error can't find the container with id 8b179ef6589eaf9b63b30caae7a4af4a15433b5ea8778a0c967bfe3b6cc91940
Oct 30 19:45:02.351158 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:02.351131    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7417f7_d341_4c53_a5b4_7239ec97c5f3.slice/crio-225cc084cc2506874a4db9eb4f7624200df1b773a7f260376f3d96d59be81d58 WatchSource:0}: Error finding container 225cc084cc2506874a4db9eb4f7624200df1b773a7f260376f3d96d59be81d58: Status 404 returned error can't find the container with id 225cc084cc2506874a4db9eb4f7624200df1b773a7f260376f3d96d59be81d58
Oct 30 19:45:02.355019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.354988    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"]
Oct 30 19:45:02.384979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.384947    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"]
Oct 30 19:45:02.389140 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:02.389100    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8510bc_b8bc_455a_8c09_7b3a7ed2dc97.slice/crio-bcfbd7eb9d6b44059825502f6cdb8877622ff33b2352ab8c0f0ab9c134dac0eb WatchSource:0}: Error finding container bcfbd7eb9d6b44059825502f6cdb8877622ff33b2352ab8c0f0ab9c134dac0eb: Status 404 returned error can't find the container with id bcfbd7eb9d6b44059825502f6cdb8877622ff33b2352ab8c0f0ab9c134dac0eb
Oct 30 19:45:02.391391 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:02.391362    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7 WatchSource:0}: Error finding container 74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7: Status 404 returned error can't find the container with id 74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7
Oct 30 19:45:02.391868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.391845    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"]
Oct 30 19:45:02.393019 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:02.392755    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93c80b0_0a57_4cbf_b218_06b9b5fe52c4.slice/crio-fac75da5add9f69fbd0196f880a20fa1ff53bdf9d0d392a19258e53e7423029b WatchSource:0}: Error finding container fac75da5add9f69fbd0196f880a20fa1ff53bdf9d0d392a19258e53e7423029b: Status 404 returned error can't find the container with id fac75da5add9f69fbd0196f880a20fa1ff53bdf9d0d392a19258e53e7423029b
Oct 30 19:45:02.397735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.397713    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"]
Oct 30 19:45:02.520489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.520448    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"]
Oct 30 19:45:02.529523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:02.529494    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:02.537742 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:02.535289    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dc9932_b47f_4181_b57f_1f021ffb67b9.slice/crio-96bc6f466eb08751337cd1b56258ef9c4e76d574b9989827852601fef6ff5eb4 WatchSource:0}: Error finding container 96bc6f466eb08751337cd1b56258ef9c4e76d574b9989827852601fef6ff5eb4: Status 404 returned error can't find the container with id 96bc6f466eb08751337cd1b56258ef9c4e76d574b9989827852601fef6ff5eb4
Oct 30 19:45:03.064288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.064237    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"]
Oct 30 19:45:03.142343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.140550    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"]
Oct 30 19:45:03.159709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.156099    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm" podUID="9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34" containerName="payload-processing" containerID="cri-o://c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386" gracePeriod=30
Oct 30 19:45:03.171565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.171413    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:03.244161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.242802    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" event={"ID":"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97","Type":"ContainerStarted","Data":"2d90c048613a556d51ffd302357dbdb4875dc366be4b8b423349df749c701157"}
Oct 30 19:45:03.244161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.242897    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" event={"ID":"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97","Type":"ContainerStarted","Data":"bcfbd7eb9d6b44059825502f6cdb8877622ff33b2352ab8c0f0ab9c134dac0eb"}
Oct 30 19:45:03.247206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.247126    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" event={"ID":"61fb7c2b-8b55-4f61-b29c-1f2093448e31","Type":"ContainerStarted","Data":"8b179ef6589eaf9b63b30caae7a4af4a15433b5ea8778a0c967bfe3b6cc91940"}
Oct 30 19:45:03.251093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.249386    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" event={"ID":"9455f850-77c8-474c-b383-cecddf457d92","Type":"ContainerStarted","Data":"d18b42de0476db4ba49eaa290ff09c2edad36852ad3b429d8e5bf22202c3f732"}
Oct 30 19:45:03.251093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.249438    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" event={"ID":"9455f850-77c8-474c-b383-cecddf457d92","Type":"ContainerStarted","Data":"9c7d9921b3ec3574e6dd0187a03cbaa2a4103b5cb1cdd28d517513aaa4a715f8"}
Oct 30 19:45:03.266663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.264958    4925 generic.go:356] "Generic (PLEG): container finished" podID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerID="a4eab4bb07534e1e2ac88f0f32ad6d3574e5e175a3e452818ae90e726b7944e0" exitCode=60
Oct 30 19:45:03.266663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.265425    4925 scope.go:122] "RemoveContainer" containerID="a4eab4bb07534e1e2ac88f0f32ad6d3574e5e175a3e452818ae90e726b7944e0"
Oct 30 19:45:03.308661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.306947    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51928b0b-54d8-4959-b115-7f091a8c8700" path="/var/lib/kubelet/pods/51928b0b-54d8-4959-b115-7f091a8c8700/volumes"
Oct 30 19:45:03.308661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.307453    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" event={"ID":"75816276-0209-464f-8a2a-bd0faca897e5","Type":"ContainerStarted","Data":"09f8fef469fd816f4792c118d41fac1e192046792e6ae89a699484125d701775"}
Oct 30 19:45:03.308661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.307533    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" event={"ID":"32e94e10-b901-4c30-8080-42b33ce1a0be","Type":"ContainerDied","Data":"a4eab4bb07534e1e2ac88f0f32ad6d3574e5e175a3e452818ae90e726b7944e0"}
Oct 30 19:45:03.308661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.307563    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" event={"ID":"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4","Type":"ContainerStarted","Data":"fac75da5add9f69fbd0196f880a20fa1ff53bdf9d0d392a19258e53e7423029b"}
Oct 30 19:45:03.308661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.307589    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" event={"ID":"2d051361-c3c7-4b15-8fb6-1462d7433649","Type":"ContainerStarted","Data":"ec7b9e2c4a3b31604bebcc93576fff49fcd2073cdb66575fa22ad424b6040b50"}
Oct 30 19:45:03.312572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.311838    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4" event={"ID":"d1c30400-2461-45d1-a898-8ad75d20331f","Type":"ContainerStarted","Data":"d652af90912978c4017493e17fc49bac2ea531bfc0a357937e04f756224a51d3"}
Oct 30 19:45:03.312572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.311894    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4" event={"ID":"d1c30400-2461-45d1-a898-8ad75d20331f","Type":"ContainerStarted","Data":"74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7"}
Oct 30 19:45:03.312977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.312584    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" podStartSLOduration=3.312569381 podStartE2EDuration="3.312569381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.276452265 +0000 UTC m=+2338.179465304" watchObservedRunningTime="2026-10-30 19:45:03.312569381 +0000 UTC m=+2338.215582419"
Oct 30 19:45:03.315998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.315547    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" event={"ID":"0a7417f7-d341-4c53-a5b4-7239ec97c5f3","Type":"ContainerStarted","Data":"225cc084cc2506874a4db9eb4f7624200df1b773a7f260376f3d96d59be81d58"}
Oct 30 19:45:03.325218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.325191    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" event={"ID":"55dc9932-b47f-4181-b57f-1f021ffb67b9","Type":"ContainerStarted","Data":"96bc6f466eb08751337cd1b56258ef9c4e76d574b9989827852601fef6ff5eb4"}
Oct 30 19:45:03.332026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.330000    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" event={"ID":"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de","Type":"ContainerStarted","Data":"ea298badf335ff1915b4b2d9598d7a68e3f0741ca9abca6d52d0462b73c4fe67"}
Oct 30 19:45:03.397475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.382344    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:03.517572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.514002    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" podStartSLOduration=3.513980169 podStartE2EDuration="3.513980169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.427883365 +0000 UTC m=+2338.330896419" watchObservedRunningTime="2026-10-30 19:45:03.513980169 +0000 UTC m=+2338.416993206"
Oct 30 19:45:03.517572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.515130    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" podStartSLOduration=3.515119497 podStartE2EDuration="3.515119497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.369438766 +0000 UTC m=+2338.272451805" watchObservedRunningTime="2026-10-30 19:45:03.515119497 +0000 UTC m=+2338.418132534"
Oct 30 19:45:03.568609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.558810    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" podStartSLOduration=3.5587886859999998 podStartE2EDuration="3.558788686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.472256105 +0000 UTC m=+2338.375269146" watchObservedRunningTime="2026-10-30 19:45:03.558788686 +0000 UTC m=+2338.461801724"
Oct 30 19:45:03.625582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.603293    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:03.689764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.660574    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4" podStartSLOduration=3.660553067 podStartE2EDuration="3.660553067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.547885708 +0000 UTC m=+2338.450898747" watchObservedRunningTime="2026-10-30 19:45:03.660553067 +0000 UTC m=+2338.563566105"
Oct 30 19:45:03.730710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.700590    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm" podUID="9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.219:9004: connect: connection refused"
Oct 30 19:45:03.789785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:03.733079    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" podStartSLOduration=3.733061293 podStartE2EDuration="3.733061293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.569584825 +0000 UTC m=+2338.472597865" watchObservedRunningTime="2026-10-30 19:45:03.733061293 +0000 UTC m=+2338.636074331"
Oct 30 19:45:04.026412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.026359    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:04.036711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.036652    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"]
Oct 30 19:45:04.101486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.101449    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7e6199-xt764"]
Oct 30 19:45:04.121903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.121869    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"]
Oct 30 19:45:04.126859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.126832    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d7d02-p927m"]
Oct 30 19:45:04.235002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.234975    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:45:04.273794 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.273761    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"]
Oct 30 19:45:04.274872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.274847    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34" containerName="payload-processing"
Oct 30 19:45:04.274872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.274870    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34" containerName="payload-processing"
Oct 30 19:45:04.275669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.275123    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34" containerName="payload-processing"
Oct 30 19:45:04.279571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.279549    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.292949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.292918    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"]
Oct 30 19:45:04.343365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.343322    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:04.345470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.345442    4925 generic.go:356] "Generic (PLEG): container finished" podID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerID="ec7b9e2c4a3b31604bebcc93576fff49fcd2073cdb66575fa22ad424b6040b50" exitCode=7
Oct 30 19:45:04.345604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.345531    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" event={"ID":"2d051361-c3c7-4b15-8fb6-1462d7433649","Type":"ContainerDied","Data":"ec7b9e2c4a3b31604bebcc93576fff49fcd2073cdb66575fa22ad424b6040b50"}
Oct 30 19:45:04.345958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.345930    4925 scope.go:122] "RemoveContainer" containerID="ec7b9e2c4a3b31604bebcc93576fff49fcd2073cdb66575fa22ad424b6040b50"
Oct 30 19:45:04.347712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.347667    4925 generic.go:356] "Generic (PLEG): container finished" podID="d1c30400-2461-45d1-a898-8ad75d20331f" containerID="d652af90912978c4017493e17fc49bac2ea531bfc0a357937e04f756224a51d3" exitCode=6
Oct 30 19:45:04.347816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.347741    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4" event={"ID":"d1c30400-2461-45d1-a898-8ad75d20331f","Type":"ContainerDied","Data":"d652af90912978c4017493e17fc49bac2ea531bfc0a357937e04f756224a51d3"}
Oct 30 19:45:04.350061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.350021    4925 generic.go:356] "Generic (PLEG): container finished" podID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerID="3e7d9339da2a143900529c0516be276683b2e5e6fb66bf805dc63680ee7056cc" exitCode=60
Oct 30 19:45:04.350169 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.350123    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" event={"ID":"0a7417f7-d341-4c53-a5b4-7239ec97c5f3","Type":"ContainerDied","Data":"3e7d9339da2a143900529c0516be276683b2e5e6fb66bf805dc63680ee7056cc"}
Oct 30 19:45:04.350635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.350420    4925 scope.go:122] "RemoveContainer" containerID="3e7d9339da2a143900529c0516be276683b2e5e6fb66bf805dc63680ee7056cc"
Oct 30 19:45:04.358252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.358220    4925 generic.go:356] "Generic (PLEG): container finished" podID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerID="c887234316cfb067e4cf66bd043f127468480aff82cac801f89d28bac89a1bf0" exitCode=60
Oct 30 19:45:04.358473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.358447    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" event={"ID":"55dc9932-b47f-4181-b57f-1f021ffb67b9","Type":"ContainerDied","Data":"c887234316cfb067e4cf66bd043f127468480aff82cac801f89d28bac89a1bf0"}
Oct 30 19:45:04.358859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.358837    4925 scope.go:122] "RemoveContainer" containerID="c887234316cfb067e4cf66bd043f127468480aff82cac801f89d28bac89a1bf0"
Oct 30 19:45:04.361258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.361234    4925 generic.go:356] "Generic (PLEG): container finished" podID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerID="ea298badf335ff1915b4b2d9598d7a68e3f0741ca9abca6d52d0462b73c4fe67" exitCode=60
Oct 30 19:45:04.361620 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.361604    4925 scope.go:122] "RemoveContainer" containerID="ea298badf335ff1915b4b2d9598d7a68e3f0741ca9abca6d52d0462b73c4fe67"
Oct 30 19:45:04.361682 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.361637    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" event={"ID":"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de","Type":"ContainerDied","Data":"ea298badf335ff1915b4b2d9598d7a68e3f0741ca9abca6d52d0462b73c4fe67"}
Oct 30 19:45:04.365711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.364604    4925 generic.go:356] "Generic (PLEG): container finished" podID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerID="2d90c048613a556d51ffd302357dbdb4875dc366be4b8b423349df749c701157" exitCode=60
Oct 30 19:45:04.365711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.364734    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" event={"ID":"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97","Type":"ContainerDied","Data":"2d90c048613a556d51ffd302357dbdb4875dc366be4b8b423349df749c701157"}
Oct 30 19:45:04.365711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.365129    4925 scope.go:122] "RemoveContainer" containerID="2d90c048613a556d51ffd302357dbdb4875dc366be4b8b423349df749c701157"
Oct 30 19:45:04.369258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.368293    4925 generic.go:356] "Generic (PLEG): container finished" podID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerID="0a2ab6e79df66a5ed04909f87cfdc110b9067a704a794609db833e81cabe662e" exitCode=60
Oct 30 19:45:04.369258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.368407    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" event={"ID":"61fb7c2b-8b55-4f61-b29c-1f2093448e31","Type":"ContainerDied","Data":"0a2ab6e79df66a5ed04909f87cfdc110b9067a704a794609db833e81cabe662e"}
Oct 30 19:45:04.369258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.368898    4925 scope.go:122] "RemoveContainer" containerID="0a2ab6e79df66a5ed04909f87cfdc110b9067a704a794609db833e81cabe662e"
Oct 30 19:45:04.372755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.371206    4925 generic.go:356] "Generic (PLEG): container finished" podID="9455f850-77c8-474c-b383-cecddf457d92" containerID="d18b42de0476db4ba49eaa290ff09c2edad36852ad3b429d8e5bf22202c3f732" exitCode=60
Oct 30 19:45:04.372755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.371288    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" event={"ID":"9455f850-77c8-474c-b383-cecddf457d92","Type":"ContainerDied","Data":"d18b42de0476db4ba49eaa290ff09c2edad36852ad3b429d8e5bf22202c3f732"}
Oct 30 19:45:04.372755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.371642    4925 scope.go:122] "RemoveContainer" containerID="d18b42de0476db4ba49eaa290ff09c2edad36852ad3b429d8e5bf22202c3f732"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.392145    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-kube-api-access-w8fgf\" (UniqueName: \"kubernetes.io/projected/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-kube-api-access-w8fgf\") pod \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\" (UID: \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\") "
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.392367    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-plugins-config-volume\") pod \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\" (UID: \"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34\") "
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.392779    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.392888    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.392916    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.392944    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.393009    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.393044    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8dzc\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-kube-api-access-b8dzc\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.394021    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-plugins-config-volume" pod "9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34" (UID: "9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.394027    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.394105    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.394257    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.394765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.394387    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.397657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.396112    4925 generic.go:356] "Generic (PLEG): container finished" podID="75816276-0209-464f-8a2a-bd0faca897e5" containerID="09f8fef469fd816f4792c118d41fac1e192046792e6ae89a699484125d701775" exitCode=60
Oct 30 19:45:04.397657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.396574    4925 scope.go:122] "RemoveContainer" containerID="09f8fef469fd816f4792c118d41fac1e192046792e6ae89a699484125d701775"
Oct 30 19:45:04.397657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.397042    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" event={"ID":"75816276-0209-464f-8a2a-bd0faca897e5","Type":"ContainerDied","Data":"09f8fef469fd816f4792c118d41fac1e192046792e6ae89a699484125d701775"}
Oct 30 19:45:04.399266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.399244    4925 generic.go:356] "Generic (PLEG): container finished" podID="9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34" containerID="c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386" exitCode=0
Oct 30 19:45:04.399359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.399347    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm" event={"ID":"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34","Type":"ContainerDied","Data":"c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386"}
Oct 30 19:45:04.399437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.399391    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm" event={"ID":"9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34","Type":"ContainerDied","Data":"592e976b0deeb2d05c5333492b58d01725a7189a42f27e96ad22befa5898281d"}
Oct 30 19:45:04.399437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.399423    4925 scope.go:122] "RemoveContainer" containerID="c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386"
Oct 30 19:45:04.399579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.399564    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"
Oct 30 19:45:04.403520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.403485    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-kube-api-access-w8fgf" pod "9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34" (UID: "9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34"). InnerVolumeSpecName "kube-api-access-w8fgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:04.412981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.404293    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" event={"ID":"32e94e10-b901-4c30-8080-42b33ce1a0be","Type":"ContainerStarted","Data":"8e40bb83729fa737bd6f6fa984e7580804962187f574e1a7b01ecc9de89451ef"}
Oct 30 19:45:04.415852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.415830    4925 generic.go:356] "Generic (PLEG): container finished" podID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerID="e2cd89a76ea175276efcb6174ee0bc39975c314f9da5e079cb10aa48d893c7d6" exitCode=60
Oct 30 19:45:04.415984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.415971    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" event={"ID":"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4","Type":"ContainerDied","Data":"e2cd89a76ea175276efcb6174ee0bc39975c314f9da5e079cb10aa48d893c7d6"}
Oct 30 19:45:04.416494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.416473    4925 scope.go:122] "RemoveContainer" containerID="e2cd89a76ea175276efcb6174ee0bc39975c314f9da5e079cb10aa48d893c7d6"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.496820    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.496894    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8dzc\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-kube-api-access-b8dzc\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.497249    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.497369    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.497435    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.497655    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.497879    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.497914    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.497946    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.498881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.498121    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8fgf\" (UniqueName: \"kubernetes.io/projected/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34-kube-api-access-w8fgf\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.501379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.501352    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.503358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.501632    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-podinfo\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.503358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.502301    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-certs\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.503515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.502638    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-credential-socket\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.503515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.502916    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-data\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.503515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.503294    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istiod-ca-cert\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.507878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.505970    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-envoy\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.521043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.520943    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8dzc\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-kube-api-access-b8dzc\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.525238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.525209    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-token\") pod \"e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") " pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.532914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.532891    4925 scope.go:122] "RemoveContainer" containerID="c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386"
Oct 30 19:45:04.537254 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:04.537182    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386\": container with ID starting with c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386 not found: ID does not exist" containerID="c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386"
Oct 30 19:45:04.537254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.537223    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386"} err="failed to get container status \"c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386\": rpc error: code = NotFound desc = could not find container \"c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386\": container with ID starting with c07539c73b0cd9eb4f45a765f0fb8bf8388b5ee8f44a0908c83b5b100d16f386 not found: ID does not exist"
Oct 30 19:45:04.578785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.576709    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:04.627989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.627951    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:04.702208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.702133    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c30400-2461-45d1-a898-8ad75d20331f-kube-api-access-vbk72\" (UniqueName: \"kubernetes.io/projected/d1c30400-2461-45d1-a898-8ad75d20331f-kube-api-access-vbk72\") pod \"d1c30400-2461-45d1-a898-8ad75d20331f\" (UID: \"d1c30400-2461-45d1-a898-8ad75d20331f\") "
Oct 30 19:45:04.702381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.702334    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c30400-2461-45d1-a898-8ad75d20331f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c30400-2461-45d1-a898-8ad75d20331f-maas-api-service-ca\") pod \"d1c30400-2461-45d1-a898-8ad75d20331f\" (UID: \"d1c30400-2461-45d1-a898-8ad75d20331f\") "
Oct 30 19:45:04.703125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.702770    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c30400-2461-45d1-a898-8ad75d20331f-maas-api-service-ca" pod "d1c30400-2461-45d1-a898-8ad75d20331f" (UID: "d1c30400-2461-45d1-a898-8ad75d20331f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:04.703125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.703093    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c30400-2461-45d1-a898-8ad75d20331f-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:04.742392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.742361    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"]
Oct 30 19:45:04.747257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.747234    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7e6199-655bb49f4d-6pjkm"]
Oct 30 19:45:04.797490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.796807    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c30400-2461-45d1-a898-8ad75d20331f-kube-api-access-vbk72" pod "d1c30400-2461-45d1-a898-8ad75d20331f" (UID: "d1c30400-2461-45d1-a898-8ad75d20331f"). InnerVolumeSpecName "kube-api-access-vbk72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:04.805465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:04.805434    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbk72\" (UniqueName: \"kubernetes.io/projected/d1c30400-2461-45d1-a898-8ad75d20331f-kube-api-access-vbk72\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:05.175841 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:05.175805    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60b132e_cabe_4f8e_8e5b_c83f998e9e84.slice/crio-9369345a519f599776f3df9d010e1f40a3b1f9715b689db76a3249bbc3dc2dac WatchSource:0}: Error finding container 9369345a519f599776f3df9d010e1f40a3b1f9715b689db76a3249bbc3dc2dac: Status 404 returned error can't find the container with id 9369345a519f599776f3df9d010e1f40a3b1f9715b689db76a3249bbc3dc2dac
Oct 30 19:45:05.176013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.175962    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"]
Oct 30 19:45:05.179092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.179053    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:05.179194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.179126    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:05.179194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.179153    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:05.351946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.350291    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:05.356523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.356087    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4292a8-327d-4cb3-ab5e-4c3f1a705482" path="/var/lib/kubelet/pods/5f4292a8-327d-4cb3-ab5e-4c3f1a705482/volumes"
Oct 30 19:45:05.357622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.357338    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34" path="/var/lib/kubelet/pods/9f50cb12-cae3-4cdd-93ca-5bb1c3b7eb34/volumes"
Oct 30 19:45:05.358545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.358510    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db391fde-68ad-4f37-97e6-de04690a45a9" path="/var/lib/kubelet/pods/db391fde-68ad-4f37-97e6-de04690a45a9/volumes"
Oct 30 19:45:05.431227 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.431190    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" event={"ID":"c60b132e-cabe-4f8e-8e5b-c83f998e9e84","Type":"ContainerStarted","Data":"76bebfa0b10ea45244d3d6c58e530524959d040dd6c680d33704d93374ff2f5f"}
Oct 30 19:45:05.431428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.431257    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" event={"ID":"c60b132e-cabe-4f8e-8e5b-c83f998e9e84","Type":"ContainerStarted","Data":"9369345a519f599776f3df9d010e1f40a3b1f9715b689db76a3249bbc3dc2dac"}
Oct 30 19:45:05.435322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.435289    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" event={"ID":"2d051361-c3c7-4b15-8fb6-1462d7433649","Type":"ContainerStarted","Data":"d0b5600f14f02d29a377ed3fe6476f061f6392b3209b6207325c69f559c57263"}
Oct 30 19:45:05.437061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.437042    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"
Oct 30 19:45:05.437207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.437185    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4" event={"ID":"d1c30400-2461-45d1-a898-8ad75d20331f","Type":"ContainerDied","Data":"74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7"}
Oct 30 19:45:05.437273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.437231    4925 scope.go:122] "RemoveContainer" containerID="d652af90912978c4017493e17fc49bac2ea531bfc0a357937e04f756224a51d3"
Oct 30 19:45:05.441019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.440991    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" event={"ID":"0a7417f7-d341-4c53-a5b4-7239ec97c5f3","Type":"ContainerStarted","Data":"d7110f91e18c484e225c8e8c80131563265bdfa09b8bf39db74c541a784ef552"}
Oct 30 19:45:05.445589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.445568    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" event={"ID":"55dc9932-b47f-4181-b57f-1f021ffb67b9","Type":"ContainerStarted","Data":"a11f210a622102b71d90e83e8f204960085d70ff6f5509373976f45e5a08dc7f"}
Oct 30 19:45:05.447818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.447799    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" event={"ID":"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de","Type":"ContainerStarted","Data":"d2c047998a031b8470018ece1f8005dc2ed8fea59eba9977302e772e0b71a47e"}
Oct 30 19:45:05.449778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.449715    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" event={"ID":"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97","Type":"ContainerStarted","Data":"0a380e33fd02e006c22d1e486eb9792d8abfe634e420d2445fe6efe32f903b7e"}
Oct 30 19:45:05.451660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.451639    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" event={"ID":"61fb7c2b-8b55-4f61-b29c-1f2093448e31","Type":"ContainerStarted","Data":"b7ccc04fa6753e13f7fa093c246b045fcee66a82c6b369b007a94762c9d19456"}
Oct 30 19:45:05.453658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.453632    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" event={"ID":"9455f850-77c8-474c-b383-cecddf457d92","Type":"ContainerStarted","Data":"ba32abe315c2e38b11d2bfb019b67de06cc3f45271204288c62ca66a428b1dce"}
Oct 30 19:45:05.458590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.458561    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" event={"ID":"75816276-0209-464f-8a2a-bd0faca897e5","Type":"ContainerStarted","Data":"90d5b9c5e6912b8ffc3e11c823d40957986a96334b4fd4f1671da9e291ab6eef"}
Oct 30 19:45:05.463269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.463243    4925 generic.go:356] "Generic (PLEG): container finished" podID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerID="8e40bb83729fa737bd6f6fa984e7580804962187f574e1a7b01ecc9de89451ef" exitCode=60
Oct 30 19:45:05.463369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.463342    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" event={"ID":"32e94e10-b901-4c30-8080-42b33ce1a0be","Type":"ContainerDied","Data":"8e40bb83729fa737bd6f6fa984e7580804962187f574e1a7b01ecc9de89451ef"}
Oct 30 19:45:05.463434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.463388    4925 scope.go:122] "RemoveContainer" containerID="a4eab4bb07534e1e2ac88f0f32ad6d3574e5e175a3e452818ae90e726b7944e0"
Oct 30 19:45:05.463553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.463507    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" podStartSLOduration=247.463489376 podStartE2EDuration="4m7.463489376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.45885115 +0000 UTC m=+2340.361864186" watchObservedRunningTime="2026-10-30 19:45:05.463489376 +0000 UTC m=+2340.366502417"
Oct 30 19:45:05.463775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.463759    4925 scope.go:122] "RemoveContainer" containerID="8e40bb83729fa737bd6f6fa984e7580804962187f574e1a7b01ecc9de89451ef"
Oct 30 19:45:05.464069 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:05.464047    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z_odh-ai-gateway-infra(32e94e10-b901-4c30-8080-42b33ce1a0be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be"
Oct 30 19:45:05.467547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.467525    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" event={"ID":"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4","Type":"ContainerStarted","Data":"7db15f9ce7c7eb1c0449b4c27e3a8e774a6d2ac7688ca2041ac10a6a99f158d6"}
Oct 30 19:45:05.490990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.490942    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" podStartSLOduration=5.490924504 podStartE2EDuration="5.490924504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.486218195 +0000 UTC m=+2340.389231233" watchObservedRunningTime="2026-10-30 19:45:05.490924504 +0000 UTC m=+2340.393937541"
Oct 30 19:45:05.508088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.508031    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" podStartSLOduration=5.508012728 podStartE2EDuration="5.508012728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.503137054 +0000 UTC m=+2340.406150098" watchObservedRunningTime="2026-10-30 19:45:05.508012728 +0000 UTC m=+2340.411025766"
Oct 30 19:45:05.565638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.565589    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" podStartSLOduration=5.565575865 podStartE2EDuration="5.565575865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.560902875 +0000 UTC m=+2340.463915911" watchObservedRunningTime="2026-10-30 19:45:05.565575865 +0000 UTC m=+2340.468588902"
Oct 30 19:45:05.580817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.580782    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"]
Oct 30 19:45:05.585748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.585717    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7e6199-29889825-rj5w4"]
Oct 30 19:45:05.628877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.628848    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:05.666569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:05.666503    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" podStartSLOduration=5.666484283 podStartE2EDuration="5.666484283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.660455 +0000 UTC m=+2340.563468040" watchObservedRunningTime="2026-10-30 19:45:05.666484283 +0000 UTC m=+2340.569497321"
Oct 30 19:45:06.033760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.033678    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:06.215714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.214734    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:06.361486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.361446    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"]
Oct 30 19:45:06.471828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.471674    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:06.511148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.511106    4925 generic.go:356] "Generic (PLEG): container finished" podID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerID="7db15f9ce7c7eb1c0449b4c27e3a8e774a6d2ac7688ca2041ac10a6a99f158d6" exitCode=60
Oct 30 19:45:06.511534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.511472    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" event={"ID":"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4","Type":"ContainerDied","Data":"7db15f9ce7c7eb1c0449b4c27e3a8e774a6d2ac7688ca2041ac10a6a99f158d6"}
Oct 30 19:45:06.511748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.511733    4925 scope.go:122] "RemoveContainer" containerID="e2cd89a76ea175276efcb6174ee0bc39975c314f9da5e079cb10aa48d893c7d6"
Oct 30 19:45:06.512592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.512576    4925 scope.go:122] "RemoveContainer" containerID="7db15f9ce7c7eb1c0449b4c27e3a8e774a6d2ac7688ca2041ac10a6a99f158d6"
Oct 30 19:45:06.513422 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.513398    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx_odh-ai-gateway-infra(e93c80b0-0a57-4cbf-b218-06b9b5fe52c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4"
Oct 30 19:45:06.526553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.526505    4925 generic.go:356] "Generic (PLEG): container finished" podID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerID="d0b5600f14f02d29a377ed3fe6476f061f6392b3209b6207325c69f559c57263" exitCode=7
Oct 30 19:45:06.526853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.526831    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" event={"ID":"2d051361-c3c7-4b15-8fb6-1462d7433649","Type":"ContainerDied","Data":"d0b5600f14f02d29a377ed3fe6476f061f6392b3209b6207325c69f559c57263"}
Oct 30 19:45:06.527631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.527612    4925 scope.go:122] "RemoveContainer" containerID="d0b5600f14f02d29a377ed3fe6476f061f6392b3209b6207325c69f559c57263"
Oct 30 19:45:06.528325 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.528300    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs_odh-ai-gateway-infra(2d051361-c3c7-4b15-8fb6-1462d7433649)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649"
Oct 30 19:45:06.545247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.545212    4925 generic.go:356] "Generic (PLEG): container finished" podID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerID="d7110f91e18c484e225c8e8c80131563265bdfa09b8bf39db74c541a784ef552" exitCode=60
Oct 30 19:45:06.546535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.546504    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" event={"ID":"0a7417f7-d341-4c53-a5b4-7239ec97c5f3","Type":"ContainerDied","Data":"d7110f91e18c484e225c8e8c80131563265bdfa09b8bf39db74c541a784ef552"}
Oct 30 19:45:06.547302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.547266    4925 scope.go:122] "RemoveContainer" containerID="d7110f91e18c484e225c8e8c80131563265bdfa09b8bf39db74c541a784ef552"
Oct 30 19:45:06.558287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.557120    4925 generic.go:356] "Generic (PLEG): container finished" podID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerID="a11f210a622102b71d90e83e8f204960085d70ff6f5509373976f45e5a08dc7f" exitCode=60
Oct 30 19:45:06.558287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.557212    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" event={"ID":"55dc9932-b47f-4181-b57f-1f021ffb67b9","Type":"ContainerDied","Data":"a11f210a622102b71d90e83e8f204960085d70ff6f5509373976f45e5a08dc7f"}
Oct 30 19:45:06.558287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.557617    4925 scope.go:122] "RemoveContainer" containerID="a11f210a622102b71d90e83e8f204960085d70ff6f5509373976f45e5a08dc7f"
Oct 30 19:45:06.558287 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.558043    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn_odh-ai-gateway-infra(55dc9932-b47f-4181-b57f-1f021ffb67b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9"
Oct 30 19:45:06.562225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.561293    4925 generic.go:356] "Generic (PLEG): container finished" podID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerID="d2c047998a031b8470018ece1f8005dc2ed8fea59eba9977302e772e0b71a47e" exitCode=60
Oct 30 19:45:06.562225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.561375    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" event={"ID":"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de","Type":"ContainerDied","Data":"d2c047998a031b8470018ece1f8005dc2ed8fea59eba9977302e772e0b71a47e"}
Oct 30 19:45:06.562225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.561757    4925 scope.go:122] "RemoveContainer" containerID="d2c047998a031b8470018ece1f8005dc2ed8fea59eba9977302e772e0b71a47e"
Oct 30 19:45:06.562225 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.561909    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv_odh-ai-gateway-infra(0a7417f7-d341-4c53-a5b4-7239ec97c5f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3"
Oct 30 19:45:06.562225 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.562158    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q_odh-ai-gateway-infra(3d2ddc65-5fb3-4c1b-bb55-41f8fec542de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de"
Oct 30 19:45:06.568812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.568638    4925 generic.go:356] "Generic (PLEG): container finished" podID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerID="0a380e33fd02e006c22d1e486eb9792d8abfe634e420d2445fe6efe32f903b7e" exitCode=60
Oct 30 19:45:06.568812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.568762    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" event={"ID":"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97","Type":"ContainerDied","Data":"0a380e33fd02e006c22d1e486eb9792d8abfe634e420d2445fe6efe32f903b7e"}
Oct 30 19:45:06.569189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.569167    4925 scope.go:122] "RemoveContainer" containerID="0a380e33fd02e006c22d1e486eb9792d8abfe634e420d2445fe6efe32f903b7e"
Oct 30 19:45:06.569657 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.569603    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh_odh-ai-gateway-infra(dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97"
Oct 30 19:45:06.572097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.571219    4925 generic.go:356] "Generic (PLEG): container finished" podID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerID="b7ccc04fa6753e13f7fa093c246b045fcee66a82c6b369b007a94762c9d19456" exitCode=60
Oct 30 19:45:06.572097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.571315    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" event={"ID":"61fb7c2b-8b55-4f61-b29c-1f2093448e31","Type":"ContainerDied","Data":"b7ccc04fa6753e13f7fa093c246b045fcee66a82c6b369b007a94762c9d19456"}
Oct 30 19:45:06.572097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.571639    4925 scope.go:122] "RemoveContainer" containerID="b7ccc04fa6753e13f7fa093c246b045fcee66a82c6b369b007a94762c9d19456"
Oct 30 19:45:06.572097 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.572052    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v_odh-ai-gateway-infra(61fb7c2b-8b55-4f61-b29c-1f2093448e31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31"
Oct 30 19:45:06.573733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.573663    4925 scope.go:122] "RemoveContainer" containerID="ec7b9e2c4a3b31604bebcc93576fff49fcd2073cdb66575fa22ad424b6040b50"
Oct 30 19:45:06.573906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.573887    4925 generic.go:356] "Generic (PLEG): container finished" podID="9455f850-77c8-474c-b383-cecddf457d92" containerID="ba32abe315c2e38b11d2bfb019b67de06cc3f45271204288c62ca66a428b1dce" exitCode=60
Oct 30 19:45:06.573985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.573970    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" event={"ID":"9455f850-77c8-474c-b383-cecddf457d92","Type":"ContainerDied","Data":"ba32abe315c2e38b11d2bfb019b67de06cc3f45271204288c62ca66a428b1dce"}
Oct 30 19:45:06.574302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.574287    4925 scope.go:122] "RemoveContainer" containerID="ba32abe315c2e38b11d2bfb019b67de06cc3f45271204288c62ca66a428b1dce"
Oct 30 19:45:06.574627 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.574609    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f_odh-ai-gateway-infra(9455f850-77c8-474c-b383-cecddf457d92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" podUID="9455f850-77c8-474c-b383-cecddf457d92"
Oct 30 19:45:06.578972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.578946    4925 generic.go:356] "Generic (PLEG): container finished" podID="75816276-0209-464f-8a2a-bd0faca897e5" containerID="90d5b9c5e6912b8ffc3e11c823d40957986a96334b4fd4f1671da9e291ab6eef" exitCode=60
Oct 30 19:45:06.579130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.579111    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" event={"ID":"75816276-0209-464f-8a2a-bd0faca897e5","Type":"ContainerDied","Data":"90d5b9c5e6912b8ffc3e11c823d40957986a96334b4fd4f1671da9e291ab6eef"}
Oct 30 19:45:06.579355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.579296    4925 scope.go:122] "RemoveContainer" containerID="90d5b9c5e6912b8ffc3e11c823d40957986a96334b4fd4f1671da9e291ab6eef"
Oct 30 19:45:06.579704 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.579664    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc_odh-ai-gateway-infra(75816276-0209-464f-8a2a-bd0faca897e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" podUID="75816276-0209-464f-8a2a-bd0faca897e5"
Oct 30 19:45:06.587679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.587655    4925 scope.go:122] "RemoveContainer" containerID="8e40bb83729fa737bd6f6fa984e7580804962187f574e1a7b01ecc9de89451ef"
Oct 30 19:45:06.588026 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:06.587922    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z_odh-ai-gateway-infra(32e94e10-b901-4c30-8080-42b33ce1a0be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be"
Oct 30 19:45:06.593209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.593185    4925 scope.go:122] "RemoveContainer" containerID="3e7d9339da2a143900529c0516be276683b2e5e6fb66bf805dc63680ee7056cc"
Oct 30 19:45:06.637719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.633358    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 19:45:06.637719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.633421    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 19:45:06.663713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.662635    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:06.800994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.800880    4925 scope.go:122] "RemoveContainer" containerID="c887234316cfb067e4cf66bd043f127468480aff82cac801f89d28bac89a1bf0"
Oct 30 19:45:06.846730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:06.846675    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:07.097050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.097017    4925 scope.go:122] "RemoveContainer" containerID="ea298badf335ff1915b4b2d9598d7a68e3f0741ca9abca6d52d0462b73c4fe67"
Oct 30 19:45:07.104717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.100218    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:07.170329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.166879    4925 scope.go:122] "RemoveContainer" containerID="2d90c048613a556d51ffd302357dbdb4875dc366be4b8b423349df749c701157"
Oct 30 19:45:07.229202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.229178    4925 scope.go:122] "RemoveContainer" containerID="0a2ab6e79df66a5ed04909f87cfdc110b9067a704a794609db833e81cabe662e"
Oct 30 19:45:07.321418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.321385    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c30400-2461-45d1-a898-8ad75d20331f" path="/var/lib/kubelet/pods/d1c30400-2461-45d1-a898-8ad75d20331f/volumes"
Oct 30 19:45:07.322203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.322178    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"]
Oct 30 19:45:07.322632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.322573    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy" containerID="cri-o://69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234" gracePeriod=30
Oct 30 19:45:07.365935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.364487    4925 scope.go:122] "RemoveContainer" containerID="d18b42de0476db4ba49eaa290ff09c2edad36852ad3b429d8e5bf22202c3f732"
Oct 30 19:45:07.387060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.387034    4925 scope.go:122] "RemoveContainer" containerID="09f8fef469fd816f4792c118d41fac1e192046792e6ae89a699484125d701775"
Oct 30 19:45:07.628986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.628867    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 19:45:07.628986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.628929    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 19:45:07.630215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.630196    4925 scope.go:122] "RemoveContainer" containerID="b7ccc04fa6753e13f7fa093c246b045fcee66a82c6b369b007a94762c9d19456"
Oct 30 19:45:07.630599 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:07.630579    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v_odh-ai-gateway-infra(61fb7c2b-8b55-4f61-b29c-1f2093448e31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31"
Oct 30 19:45:07.633610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.633573    4925 scope.go:122] "RemoveContainer" containerID="ba32abe315c2e38b11d2bfb019b67de06cc3f45271204288c62ca66a428b1dce"
Oct 30 19:45:07.633988 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:07.633962    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f_odh-ai-gateway-infra(9455f850-77c8-474c-b383-cecddf457d92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" podUID="9455f850-77c8-474c-b383-cecddf457d92"
Oct 30 19:45:07.637935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.637916    4925 scope.go:122] "RemoveContainer" containerID="90d5b9c5e6912b8ffc3e11c823d40957986a96334b4fd4f1671da9e291ab6eef"
Oct 30 19:45:07.638389 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:07.638368    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc_odh-ai-gateway-infra(75816276-0209-464f-8a2a-bd0faca897e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" podUID="75816276-0209-464f-8a2a-bd0faca897e5"
Oct 30 19:45:07.641155 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.641135    4925 scope.go:122] "RemoveContainer" containerID="7db15f9ce7c7eb1c0449b4c27e3a8e774a6d2ac7688ca2041ac10a6a99f158d6"
Oct 30 19:45:07.641487 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:07.641465    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx_odh-ai-gateway-infra(e93c80b0-0a57-4cbf-b218-06b9b5fe52c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4"
Oct 30 19:45:07.645063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.645040    4925 scope.go:122] "RemoveContainer" containerID="d0b5600f14f02d29a377ed3fe6476f061f6392b3209b6207325c69f559c57263"
Oct 30 19:45:07.645420 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:07.645389    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs_odh-ai-gateway-infra(2d051361-c3c7-4b15-8fb6-1462d7433649)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649"
Oct 30 19:45:07.648032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.647987    4925 scope.go:122] "RemoveContainer" containerID="d7110f91e18c484e225c8e8c80131563265bdfa09b8bf39db74c541a784ef552"
Oct 30 19:45:07.648281 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:07.648259    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv_odh-ai-gateway-infra(0a7417f7-d341-4c53-a5b4-7239ec97c5f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3"
Oct 30 19:45:07.650010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.649996    4925 scope.go:122] "RemoveContainer" containerID="a11f210a622102b71d90e83e8f204960085d70ff6f5509373976f45e5a08dc7f"
Oct 30 19:45:07.651334 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:07.651309    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn_odh-ai-gateway-infra(55dc9932-b47f-4181-b57f-1f021ffb67b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9"
Oct 30 19:45:07.653767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.653746    4925 scope.go:122] "RemoveContainer" containerID="d2c047998a031b8470018ece1f8005dc2ed8fea59eba9977302e772e0b71a47e"
Oct 30 19:45:07.654514 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:07.654490    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q_odh-ai-gateway-infra(3d2ddc65-5fb3-4c1b-bb55-41f8fec542de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de"
Oct 30 19:45:07.656711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:07.656666    4925 scope.go:122] "RemoveContainer" containerID="0a380e33fd02e006c22d1e486eb9792d8abfe634e420d2445fe6efe32f903b7e"
Oct 30 19:45:07.657036 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:07.657013    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh_odh-ai-gateway-infra(dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97"
Oct 30 19:45:08.345734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:08.345673    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:08.630171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:08.630080    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 19:45:08.630171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:08.630140    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 19:45:09.250267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:09.250219    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:09.429240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:09.428923    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:09.637167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:09.637070    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 19:45:09.637799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:09.637765    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 19:45:09.660709 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:09.641858    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:10.097724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:10.091043    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:10.325043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:10.325011    4925 scope.go:122] "RemoveContainer" containerID="2ea39bb13c635e9b68f10baf799b1b13b7671fafd51daa3dd94d08297e93aa24"
Oct 30 19:45:10.458190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:10.458164    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:10.644769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:10.632637    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 19:45:10.644769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:10.632738    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 19:45:11.363789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.362864    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8" podUID="4e287e06-4476-46bb-ae77-ee3e208f06d9" containerName="payload-pre-processing" containerID="cri-o://a6dedfb039eee34d7013128e29a131070473f7f5de99af18134e262e5b78a642" gracePeriod=25
Oct 30 19:45:11.629036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.628958    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 19:45:11.629195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.629026    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 19:45:11.749932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.747652    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:11.835849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.835796    4925 generic.go:356] "Generic (PLEG): container finished" podID="4e287e06-4476-46bb-ae77-ee3e208f06d9" containerID="a6dedfb039eee34d7013128e29a131070473f7f5de99af18134e262e5b78a642" exitCode=0
Oct 30 19:45:11.836092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.836066    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8" event={"ID":"4e287e06-4476-46bb-ae77-ee3e208f06d9","Type":"ContainerDied","Data":"a6dedfb039eee34d7013128e29a131070473f7f5de99af18134e262e5b78a642"}
Oct 30 19:45:11.879835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.879770    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:45:11.935350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.933517    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:11.951440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.951404    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"]
Oct 30 19:45:11.953727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.952749    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e287e06-4476-46bb-ae77-ee3e208f06d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e287e06-4476-46bb-ae77-ee3e208f06d9-plugins-config-volume\") pod \"4e287e06-4476-46bb-ae77-ee3e208f06d9\" (UID: \"4e287e06-4476-46bb-ae77-ee3e208f06d9\") "
Oct 30 19:45:11.953727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.953006    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e287e06-4476-46bb-ae77-ee3e208f06d9-kube-api-access-f6dpp\" (UniqueName: \"kubernetes.io/projected/4e287e06-4476-46bb-ae77-ee3e208f06d9-kube-api-access-f6dpp\") pod \"4e287e06-4476-46bb-ae77-ee3e208f06d9\" (UID: \"4e287e06-4476-46bb-ae77-ee3e208f06d9\") "
Oct 30 19:45:11.954565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.954524    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e287e06-4476-46bb-ae77-ee3e208f06d9-plugins-config-volume" pod "4e287e06-4476-46bb-ae77-ee3e208f06d9" (UID: "4e287e06-4476-46bb-ae77-ee3e208f06d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:11.955182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.955060    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e287e06-4476-46bb-ae77-ee3e208f06d9-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:11.955182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.955104    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy" containerID="cri-o://ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3" gracePeriod=30
Oct 30 19:45:11.960499 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:11.960460    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e287e06-4476-46bb-ae77-ee3e208f06d9-kube-api-access-f6dpp" pod "4e287e06-4476-46bb-ae77-ee3e208f06d9" (UID: "4e287e06-4476-46bb-ae77-ee3e208f06d9"). InnerVolumeSpecName "kube-api-access-f6dpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:12.081711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.073738    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6dpp\" (UniqueName: \"kubernetes.io/projected/4e287e06-4476-46bb-ae77-ee3e208f06d9-kube-api-access-f6dpp\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.111711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.110398    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:12.576013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.575977    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:12.636190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.630097    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 19:45:12.636190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.630170    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 19:45:12.844936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.844910    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:45:12.848835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.848807    4925 generic.go:356] "Generic (PLEG): container finished" podID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerID="69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234" exitCode=0
Oct 30 19:45:12.848944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.848851    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"
Oct 30 19:45:12.848944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.848907    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" event={"ID":"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174","Type":"ContainerDied","Data":"69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234"}
Oct 30 19:45:12.849033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.848948    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk" event={"ID":"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174","Type":"ContainerDied","Data":"83fa5caac0c5f53e3f96d0a4dd3929ca199e20d617e8679b67a0aa7365e6233f"}
Oct 30 19:45:12.849033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.848969    4925 scope.go:122] "RemoveContainer" containerID="69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234"
Oct 30 19:45:12.850815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.850795    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"
Oct 30 19:45:12.850906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.850824    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8" event={"ID":"4e287e06-4476-46bb-ae77-ee3e208f06d9","Type":"ContainerDied","Data":"834925e8cee94a00d95d7ecc7dce521d88fe14c6c8d69d15975e5c5d821a22fc"}
Oct 30 19:45:12.866433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.866414    4925 scope.go:122] "RemoveContainer" containerID="69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234"
Oct 30 19:45:12.866753 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:12.866725    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234\": container with ID starting with 69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234 not found: ID does not exist" containerID="69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234"
Oct 30 19:45:12.866831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.866763    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234"} err="failed to get container status \"69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234\": rpc error: code = NotFound desc = could not find container \"69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234\": container with ID starting with 69ac48486c52ec7ef10c597eaa8779d41125080842595c0a4a9e47ffc0968234 not found: ID does not exist"
Oct 30 19:45:12.866831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.866780    4925 scope.go:122] "RemoveContainer" containerID="a6dedfb039eee34d7013128e29a131070473f7f5de99af18134e262e5b78a642"
Oct 30 19:45:12.886756 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:12.885142    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:12.931579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.931532    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"]
Oct 30 19:45:12.940436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.940392    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-65949fbbd7-jphs8"]
Oct 30 19:45:12.944705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.944661    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istiod-ca-cert\") pod \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") "
Oct 30 19:45:12.944886 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.944865    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-envoy\") pod \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") "
Oct 30 19:45:12.945111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945091    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-kube-api-access-2dtzf\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-kube-api-access-2dtzf\") pod \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") "
Oct 30 19:45:12.945241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945223    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-socket\") pod \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") "
Oct 30 19:45:12.945306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945264    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-data\") pod \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") "
Oct 30 19:45:12.945367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945359    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-credential-socket\") pod \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") "
Oct 30 19:45:12.945420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945358    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istiod-ca-cert" pod "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" (UID: "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:12.945420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945375    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-certs\") pod \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") "
Oct 30 19:45:12.945534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945472    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-podinfo\") pod \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") "
Oct 30 19:45:12.945634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945612    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-token\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-token\") pod \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\" (UID: \"5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174\") "
Oct 30 19:45:12.945763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945740    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-data" pod "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" (UID: "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.945908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.945741    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-socket" pod "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" (UID: "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.946342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.946269    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-certs" pod "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" (UID: "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.946631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.946592    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.946631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.946627    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.946804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.946643    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.946804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.946655    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.946935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.946914    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-credential-socket" pod "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" (UID: "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.948828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.948807    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-token" pod "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" (UID: "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:12.949478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.949455    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-envoy" pod "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" (UID: "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:12.949617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.949600    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-kube-api-access-2dtzf" pod "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" (UID: "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174"). InnerVolumeSpecName "kube-api-access-2dtzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:12.950609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:12.950590    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-podinfo" pod "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" (UID: "5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:13.048669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.048581    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:13.048669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.048625    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtzf\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-kube-api-access-2dtzf\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:13.048669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.048641    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:13.048669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.048655    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:13.048669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.048669    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:13.176315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.176277    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"]
Oct 30 19:45:13.184334 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.184296    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-wg8mk"]
Oct 30 19:45:13.269871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.269833    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e287e06-4476-46bb-ae77-ee3e208f06d9" path="/var/lib/kubelet/pods/4e287e06-4476-46bb-ae77-ee3e208f06d9/volumes"
Oct 30 19:45:13.272522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.270680    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" path="/var/lib/kubelet/pods/5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174/volumes"
Oct 30 19:45:13.330445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.330326    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:13.514712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.507710    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:13.747731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.740340    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:13.764412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.764378    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:13.766929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.765454    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:13.770982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.770955    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.876007    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"]
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.876887    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy"
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.876903    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy"
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.876935    4925 cpu_manager.go:414] "containerMap: removing container" podUID="4e287e06-4476-46bb-ae77-ee3e208f06d9" containerName="payload-pre-processing"
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.876940    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e287e06-4476-46bb-ae77-ee3e208f06d9" containerName="payload-pre-processing"
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.876965    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d1c30400-2461-45d1-a898-8ad75d20331f" containerName="cleanup"
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.876970    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c30400-2461-45d1-a898-8ad75d20331f" containerName="cleanup"
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.877079    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c30400-2461-45d1-a898-8ad75d20331f" containerName="cleanup"
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.877095    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8cb6ca-1b2a-4cef-95b4-c13f7eba7174" containerName="istio-proxy"
Oct 30 19:45:13.881219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.877106    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e287e06-4476-46bb-ae77-ee3e208f06d9" containerName="payload-pre-processing"
Oct 30 19:45:13.883516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.883489    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:13.888104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.888079    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"]
Oct 30 19:45:13.889704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.889666    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:13.984828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.984798    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5nq\" (UniqueName: \"kubernetes.io/projected/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-kube-api-access-pv5nq\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:13.985114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.985091    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:13.985213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:13.985134    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:14.087394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.087268    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:14.087394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.087371    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:14.087668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.087651    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5nq\" (UniqueName: \"kubernetes.io/projected/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-kube-api-access-pv5nq\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:14.088070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.088047    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:14.089999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.089974    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:14.101677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.101651    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5nq\" (UniqueName: \"kubernetes.io/projected/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-kube-api-access-pv5nq\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:14.209893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.209863    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:14.498303 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:14.498273    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200f584f_edfd_4c92_8b9d_7251fbdc1b0e.slice/crio-2d136e1e7d2c03a6261a18b875ee6e46416366c74451e82f575b00388e887f98 WatchSource:0}: Error finding container 2d136e1e7d2c03a6261a18b875ee6e46416366c74451e82f575b00388e887f98: Status 404 returned error can't find the container with id 2d136e1e7d2c03a6261a18b875ee6e46416366c74451e82f575b00388e887f98
Oct 30 19:45:14.498449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.498340    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"]
Oct 30 19:45:14.649761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.649724    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:14.893302 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:14.893263    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:14.893823 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:14.893327    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:14.896420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:14.896384    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8" event={"ID":"200f584f-edfd-4c92-8b9d-7251fbdc1b0e","Type":"ContainerStarted","Data":"2d136e1e7d2c03a6261a18b875ee6e46416366c74451e82f575b00388e887f98"}
Oct 30 19:45:15.035208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.035173    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"]
Oct 30 19:45:15.043285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.043258    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf9544c0-m99lt"]
Oct 30 19:45:15.176873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.176789    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:15.276320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.276287    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a04830de-ec49-44a4-9de6-78743d480279" path="/var/lib/kubelet/pods/a04830de-ec49-44a4-9de6-78743d480279/volumes"
Oct 30 19:45:15.587443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.587413    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"]
Oct 30 19:45:15.592257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.592239    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:15.599185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.599159    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"]
Oct 30 19:45:15.890768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.875522    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:15.890768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.875969    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:15.890768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:15.876124    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gdq\" (UniqueName: \"kubernetes.io/projected/9dc40c87-1067-48c8-9259-b8d7581369f5-kube-api-access-f4gdq\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:16.028276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.025382    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:16.028276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.025544    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gdq\" (UniqueName: \"kubernetes.io/projected/9dc40c87-1067-48c8-9259-b8d7581369f5-kube-api-access-f4gdq\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:16.028276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.025622    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:16.028276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.025983    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:16.028276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.026649    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:16.042419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.042385    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gdq\" (UniqueName: \"kubernetes.io/projected/9dc40c87-1067-48c8-9259-b8d7581369f5-kube-api-access-f4gdq\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:16.045197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.045171    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:16.047186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.047159    4925 generic.go:356] "Generic (PLEG): container finished" podID="200f584f-edfd-4c92-8b9d-7251fbdc1b0e" containerID="88025e997075cd9cd4a3bd5f466b234fee45c91fdb723341871196c038d3f9a4" exitCode=0
Oct 30 19:45:16.047186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.047227    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8" event={"ID":"200f584f-edfd-4c92-8b9d-7251fbdc1b0e","Type":"ContainerDied","Data":"88025e997075cd9cd4a3bd5f466b234fee45c91fdb723341871196c038d3f9a4"}
Oct 30 19:45:16.209795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.209722    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:16.256021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.255991    4925 scope.go:122] "RemoveContainer" containerID="7db15f9ce7c7eb1c0449b4c27e3a8e774a6d2ac7688ca2041ac10a6a99f158d6"
Oct 30 19:45:16.499873 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:16.499844    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc40c87_1067_48c8_9259_b8d7581369f5.slice/crio-05e44f905cac10628e9c0065b38accf38659a9c84130886fbe3aa9418f520a89 WatchSource:0}: Error finding container 05e44f905cac10628e9c0065b38accf38659a9c84130886fbe3aa9418f520a89: Status 404 returned error can't find the container with id 05e44f905cac10628e9c0065b38accf38659a9c84130886fbe3aa9418f520a89
Oct 30 19:45:16.500045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.500020    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"]
Oct 30 19:45:16.793765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:16.793306    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:17.058256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.058168    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5" event={"ID":"9dc40c87-1067-48c8-9259-b8d7581369f5","Type":"ContainerStarted","Data":"173d8955c4c829113a4e349d9031339f5c200cae26dede90dab2530a5671847c"}
Oct 30 19:45:17.058256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.058221    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5" event={"ID":"9dc40c87-1067-48c8-9259-b8d7581369f5","Type":"ContainerStarted","Data":"05e44f905cac10628e9c0065b38accf38659a9c84130886fbe3aa9418f520a89"}
Oct 30 19:45:17.060467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.060438    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" event={"ID":"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4","Type":"ContainerStarted","Data":"8ae657c30fba5efb7c8f10f13f65ffb21ea6ff17e38b4ce009da931ae4c0b463"}
Oct 30 19:45:17.081933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.081887    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5" podStartSLOduration=2.081871737 podStartE2EDuration="2.081871737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:17.077443359 +0000 UTC m=+2351.980456399" watchObservedRunningTime="2026-10-30 19:45:17.081871737 +0000 UTC m=+2351.984884752"
Oct 30 19:45:17.260853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.260826    4925 scope.go:122] "RemoveContainer" containerID="90d5b9c5e6912b8ffc3e11c823d40957986a96334b4fd4f1671da9e291ab6eef"
Oct 30 19:45:17.261797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.261775    4925 scope.go:122] "RemoveContainer" containerID="ba32abe315c2e38b11d2bfb019b67de06cc3f45271204288c62ca66a428b1dce"
Oct 30 19:45:17.346709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.346664    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:17.363243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.363223    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:45:17.454537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.454448    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:17.560526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.560498    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7febe178-dee4-4add-9355-5da5c8dfc1ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7febe178-dee4-4add-9355-5da5c8dfc1ee-istiod-ca-cert\") pod \"7febe178-dee4-4add-9355-5da5c8dfc1ee\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") "
Oct 30 19:45:17.560667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.560542    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-data\") pod \"7febe178-dee4-4add-9355-5da5c8dfc1ee\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") "
Oct 30 19:45:17.560757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.560671    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-credential-socket\") pod \"7febe178-dee4-4add-9355-5da5c8dfc1ee\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") "
Oct 30 19:45:17.560757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.560742    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-kube-api-access-pb9m4\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-kube-api-access-pb9m4\") pod \"7febe178-dee4-4add-9355-5da5c8dfc1ee\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") "
Oct 30 19:45:17.560882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.560861    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-token\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-token\") pod \"7febe178-dee4-4add-9355-5da5c8dfc1ee\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") "
Oct 30 19:45:17.561008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.560897    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-openshift-ca\") pod \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") "
Oct 30 19:45:17.561008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.560929    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-certs\") pod \"7febe178-dee4-4add-9355-5da5c8dfc1ee\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") "
Oct 30 19:45:17.561008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.560952    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-data" pod "7febe178-dee4-4add-9355-5da5c8dfc1ee" (UID: "7febe178-dee4-4add-9355-5da5c8dfc1ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.561218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561012    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-envoy\") pod \"7febe178-dee4-4add-9355-5da5c8dfc1ee\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") "
Oct 30 19:45:17.561218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561020    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-credential-socket" pod "7febe178-dee4-4add-9355-5da5c8dfc1ee" (UID: "7febe178-dee4-4add-9355-5da5c8dfc1ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.561218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561085    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7febe178-dee4-4add-9355-5da5c8dfc1ee-istiod-ca-cert" pod "7febe178-dee4-4add-9355-5da5c8dfc1ee" (UID: "7febe178-dee4-4add-9355-5da5c8dfc1ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:17.561218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561097    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-socket\") pod \"7febe178-dee4-4add-9355-5da5c8dfc1ee\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") "
Oct 30 19:45:17.561218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561170    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-xks-ca\") pod \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") "
Oct 30 19:45:17.561533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561227    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-certs" pod "7febe178-dee4-4add-9355-5da5c8dfc1ee" (UID: "7febe178-dee4-4add-9355-5da5c8dfc1ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.561533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561233    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-podinfo\") pod \"7febe178-dee4-4add-9355-5da5c8dfc1ee\" (UID: \"7febe178-dee4-4add-9355-5da5c8dfc1ee\") "
Oct 30 19:45:17.561533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561284    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-openshift-ca" pod "200f584f-edfd-4c92-8b9d-7251fbdc1b0e" (UID: "200f584f-edfd-4c92-8b9d-7251fbdc1b0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:17.561533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561368    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-socket" pod "7febe178-dee4-4add-9355-5da5c8dfc1ee" (UID: "7febe178-dee4-4add-9355-5da5c8dfc1ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.561533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.561298    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-kube-api-access-pv5nq\" (UniqueName: \"kubernetes.io/projected/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-kube-api-access-pv5nq\") pod \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\" (UID: \"200f584f-edfd-4c92-8b9d-7251fbdc1b0e\") "
Oct 30 19:45:17.562546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.562508    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.562546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.562542    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7febe178-dee4-4add-9355-5da5c8dfc1ee-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.563285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.562557    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.563285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.562570    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.563285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.562583    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.563285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.562597    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.563555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.563540    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-xks-ca" pod "200f584f-edfd-4c92-8b9d-7251fbdc1b0e" (UID: "200f584f-edfd-4c92-8b9d-7251fbdc1b0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:17.563961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.563941    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-token" pod "7febe178-dee4-4add-9355-5da5c8dfc1ee" (UID: "7febe178-dee4-4add-9355-5da5c8dfc1ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:17.564154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.564134    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-kube-api-access-pb9m4" pod "7febe178-dee4-4add-9355-5da5c8dfc1ee" (UID: "7febe178-dee4-4add-9355-5da5c8dfc1ee"). InnerVolumeSpecName "kube-api-access-pb9m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:17.564246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.564151    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-podinfo" pod "7febe178-dee4-4add-9355-5da5c8dfc1ee" (UID: "7febe178-dee4-4add-9355-5da5c8dfc1ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:17.564246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.564164    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-kube-api-access-pv5nq" pod "200f584f-edfd-4c92-8b9d-7251fbdc1b0e" (UID: "200f584f-edfd-4c92-8b9d-7251fbdc1b0e"). InnerVolumeSpecName "kube-api-access-pv5nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:17.564500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.564482    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-envoy" pod "7febe178-dee4-4add-9355-5da5c8dfc1ee" (UID: "7febe178-dee4-4add-9355-5da5c8dfc1ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.663955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.663924    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.663955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.663948    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.663955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.663958    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv5nq\" (UniqueName: \"kubernetes.io/projected/200f584f-edfd-4c92-8b9d-7251fbdc1b0e-kube-api-access-pv5nq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.664178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.663967    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb9m4\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-kube-api-access-pb9m4\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.664178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.663977    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.664178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:17.663987    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7febe178-dee4-4add-9355-5da5c8dfc1ee-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:18.075876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.075556    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:18.114445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.114403    4925 generic.go:356] "Generic (PLEG): container finished" podID="9dc40c87-1067-48c8-9259-b8d7581369f5" containerID="173d8955c4c829113a4e349d9031339f5c200cae26dede90dab2530a5671847c" exitCode=0
Oct 30 19:45:18.114623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.114537    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5" event={"ID":"9dc40c87-1067-48c8-9259-b8d7581369f5","Type":"ContainerDied","Data":"173d8955c4c829113a4e349d9031339f5c200cae26dede90dab2530a5671847c"}
Oct 30 19:45:18.116658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.116632    4925 generic.go:356] "Generic (PLEG): container finished" podID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerID="ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3" exitCode=0
Oct 30 19:45:18.116812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.116732    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" event={"ID":"7febe178-dee4-4add-9355-5da5c8dfc1ee","Type":"ContainerDied","Data":"ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3"}
Oct 30 19:45:18.116812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.116754    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"
Oct 30 19:45:18.116812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.116782    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8" event={"ID":"7febe178-dee4-4add-9355-5da5c8dfc1ee","Type":"ContainerDied","Data":"633cf2766df82ee6f6efea4216baf7918dcfab74e258b09ad3fe3630a3e822db"}
Oct 30 19:45:18.116812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.116811    4925 scope.go:122] "RemoveContainer" containerID="ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3"
Oct 30 19:45:18.119277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.119249    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" event={"ID":"9455f850-77c8-474c-b383-cecddf457d92","Type":"ContainerStarted","Data":"124e24b8bffdf69bd92910ea4d58fa240a497fee0e68e744903d66f49a5d26fb"}
Oct 30 19:45:18.120361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.120333    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"]
Oct 30 19:45:18.121554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.121413    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" event={"ID":"75816276-0209-464f-8a2a-bd0faca897e5","Type":"ContainerStarted","Data":"1ad1b9c60970aec7ed73e15e2e4bd55d8cb1d0735d4de7dccae06434bcd342ef"}
Oct 30 19:45:18.124064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.124045    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"
Oct 30 19:45:18.124236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.124081    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8" event={"ID":"200f584f-edfd-4c92-8b9d-7251fbdc1b0e","Type":"ContainerDied","Data":"2d136e1e7d2c03a6261a18b875ee6e46416366c74451e82f575b00388e887f98"}
Oct 30 19:45:18.124236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.124128    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d136e1e7d2c03a6261a18b875ee6e46416366c74451e82f575b00388e887f98"
Oct 30 19:45:18.131152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.131129    4925 generic.go:356] "Generic (PLEG): container finished" podID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerID="8ae657c30fba5efb7c8f10f13f65ffb21ea6ff17e38b4ce009da931ae4c0b463" exitCode=60
Oct 30 19:45:18.131247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.131195    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" event={"ID":"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4","Type":"ContainerDied","Data":"8ae657c30fba5efb7c8f10f13f65ffb21ea6ff17e38b4ce009da931ae4c0b463"}
Oct 30 19:45:18.131761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.131670    4925 scope.go:122] "RemoveContainer" containerID="bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262"
Oct 30 19:45:18.146932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.146911    4925 scope.go:122] "RemoveContainer" containerID="ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3"
Oct 30 19:45:18.147286 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:18.147260    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3\": container with ID starting with ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3 not found: ID does not exist" containerID="ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3"
Oct 30 19:45:18.147357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.147301    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3"} err="failed to get container status \"ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3\": rpc error: code = NotFound desc = could not find container \"ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3\": container with ID starting with ce585ca9a9f0f11a1c56d7c5a90120c16ebe9fb265304bd8d913787d305810f3 not found: ID does not exist"
Oct 30 19:45:18.147357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.147326    4925 scope.go:122] "RemoveContainer" containerID="bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262"
Oct 30 19:45:18.147654 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:18.147617    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262\": container with ID starting with bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262 not found: ID does not exist" containerID="bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262"
Oct 30 19:45:18.147745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.147654    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262"} err="failed to get container status \"bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262\": rpc error: code = NotFound desc = could not find container \"bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262\": container with ID starting with bc73d6942d6d075fff376e64e36fded6210f1627b86efd8d60dca25228192262 not found: ID does not exist"
Oct 30 19:45:18.147745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.147677    4925 scope.go:122] "RemoveContainer" containerID="7db15f9ce7c7eb1c0449b4c27e3a8e774a6d2ac7688ca2041ac10a6a99f158d6"
Oct 30 19:45:18.181812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.181784    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"]
Oct 30 19:45:18.187133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.187106    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7e6199-openshift-default-5cdcfcd5d5-twjq8"]
Oct 30 19:45:18.262185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.262157    4925 scope.go:122] "RemoveContainer" containerID="0a380e33fd02e006c22d1e486eb9792d8abfe634e420d2445fe6efe32f903b7e"
Oct 30 19:45:18.262527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.262503    4925 scope.go:122] "RemoveContainer" containerID="a11f210a622102b71d90e83e8f204960085d70ff6f5509373976f45e5a08dc7f"
Oct 30 19:45:18.262674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.262656    4925 scope.go:122] "RemoveContainer" containerID="d7110f91e18c484e225c8e8c80131563265bdfa09b8bf39db74c541a784ef552"
Oct 30 19:45:18.348553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.348499    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:18.378490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.378460    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-kube-api-access-k6b4n\" (UniqueName: \"kubernetes.io/projected/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-kube-api-access-k6b4n\") pod \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\" (UID: \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\") "
Oct 30 19:45:18.378631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.378568    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-maas-api-service-ca\") pod \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\" (UID: \"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4\") "
Oct 30 19:45:18.379416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.379384    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-maas-api-service-ca" pod "e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" (UID: "e93c80b0-0a57-4cbf-b218-06b9b5fe52c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:18.382012 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.381987    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-kube-api-access-k6b4n" pod "e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" (UID: "e93c80b0-0a57-4cbf-b218-06b9b5fe52c4"). InnerVolumeSpecName "kube-api-access-k6b4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:18.481134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.481096    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:18.481134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:18.481124    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6b4n\" (UniqueName: \"kubernetes.io/projected/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4-kube-api-access-k6b4n\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.042282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.042202    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:19.142670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.142638    4925 generic.go:356] "Generic (PLEG): container finished" podID="9455f850-77c8-474c-b383-cecddf457d92" containerID="124e24b8bffdf69bd92910ea4d58fa240a497fee0e68e744903d66f49a5d26fb" exitCode=60
Oct 30 19:45:19.143136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.142719    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" event={"ID":"9455f850-77c8-474c-b383-cecddf457d92","Type":"ContainerDied","Data":"124e24b8bffdf69bd92910ea4d58fa240a497fee0e68e744903d66f49a5d26fb"}
Oct 30 19:45:19.143136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.142788    4925 scope.go:122] "RemoveContainer" containerID="ba32abe315c2e38b11d2bfb019b67de06cc3f45271204288c62ca66a428b1dce"
Oct 30 19:45:19.143270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.143252    4925 scope.go:122] "RemoveContainer" containerID="124e24b8bffdf69bd92910ea4d58fa240a497fee0e68e744903d66f49a5d26fb"
Oct 30 19:45:19.143890 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:19.143761    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f_odh-ai-gateway-infra(9455f850-77c8-474c-b383-cecddf457d92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" podUID="9455f850-77c8-474c-b383-cecddf457d92"
Oct 30 19:45:19.144917 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.144888    4925 generic.go:356] "Generic (PLEG): container finished" podID="75816276-0209-464f-8a2a-bd0faca897e5" containerID="1ad1b9c60970aec7ed73e15e2e4bd55d8cb1d0735d4de7dccae06434bcd342ef" exitCode=60
Oct 30 19:45:19.145025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.144965    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" event={"ID":"75816276-0209-464f-8a2a-bd0faca897e5","Type":"ContainerDied","Data":"1ad1b9c60970aec7ed73e15e2e4bd55d8cb1d0735d4de7dccae06434bcd342ef"}
Oct 30 19:45:19.145372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.145353    4925 scope.go:122] "RemoveContainer" containerID="1ad1b9c60970aec7ed73e15e2e4bd55d8cb1d0735d4de7dccae06434bcd342ef"
Oct 30 19:45:19.145897 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:19.145872    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc_odh-ai-gateway-infra(75816276-0209-464f-8a2a-bd0faca897e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" podUID="75816276-0209-464f-8a2a-bd0faca897e5"
Oct 30 19:45:19.146497 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.146481    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"
Oct 30 19:45:19.146571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.146497    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx" event={"ID":"e93c80b0-0a57-4cbf-b218-06b9b5fe52c4","Type":"ContainerDied","Data":"fac75da5add9f69fbd0196f880a20fa1ff53bdf9d0d392a19258e53e7423029b"}
Oct 30 19:45:19.148545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.148526    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" event={"ID":"0a7417f7-d341-4c53-a5b4-7239ec97c5f3","Type":"ContainerStarted","Data":"4dc2cbbb188054582ed49bbd744b532a3b711cbed79b22ff8777ce168c157cf8"}
Oct 30 19:45:19.150670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.150650    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" event={"ID":"55dc9932-b47f-4181-b57f-1f021ffb67b9","Type":"ContainerStarted","Data":"77bf37ccd1f2ac200bdc67e770f6259bf89ff3fc45cc7f72365c3a10e7e82e71"}
Oct 30 19:45:19.152874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.152822    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" event={"ID":"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97","Type":"ContainerStarted","Data":"d65cc905c98be0c308d3e105996e3cec2d28dd983797babd62b5af0b02988ff0"}
Oct 30 19:45:19.159097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.159080    4925 scope.go:122] "RemoveContainer" containerID="90d5b9c5e6912b8ffc3e11c823d40957986a96334b4fd4f1671da9e291ab6eef"
Oct 30 19:45:19.172940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.172912    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"]
Oct 30 19:45:19.179663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.179636    4925 scope.go:122] "RemoveContainer" containerID="8ae657c30fba5efb7c8f10f13f65ffb21ea6ff17e38b4ce009da931ae4c0b463"
Oct 30 19:45:19.229506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.229474    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"]
Oct 30 19:45:19.276292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.273759    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" path="/var/lib/kubelet/pods/7febe178-dee4-4add-9355-5da5c8dfc1ee/volumes"
Oct 30 19:45:19.276292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.274362    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"]
Oct 30 19:45:19.276292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.274400    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2d5d53-29889825-6kgvx"]
Oct 30 19:45:19.276292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.275053    4925 scope.go:122] "RemoveContainer" containerID="d0b5600f14f02d29a377ed3fe6476f061f6392b3209b6207325c69f559c57263"
Oct 30 19:45:19.277058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.276743    4925 scope.go:122] "RemoveContainer" containerID="8e40bb83729fa737bd6f6fa984e7580804962187f574e1a7b01ecc9de89451ef"
Oct 30 19:45:19.277517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.277213    4925 scope.go:122] "RemoveContainer" containerID="b7ccc04fa6753e13f7fa093c246b045fcee66a82c6b369b007a94762c9d19456"
Oct 30 19:45:19.386762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.386737    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:19.394889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.394865    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc40c87-1067-48c8-9259-b8d7581369f5-kube-api-access-f4gdq\" (UniqueName: \"kubernetes.io/projected/9dc40c87-1067-48c8-9259-b8d7581369f5-kube-api-access-f4gdq\") pod \"9dc40c87-1067-48c8-9259-b8d7581369f5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") "
Oct 30 19:45:19.395025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.394908    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-openshift-ca\") pod \"9dc40c87-1067-48c8-9259-b8d7581369f5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") "
Oct 30 19:45:19.395326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.395173    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-xks-ca\") pod \"9dc40c87-1067-48c8-9259-b8d7581369f5\" (UID: \"9dc40c87-1067-48c8-9259-b8d7581369f5\") "
Oct 30 19:45:19.395608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.395530    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-openshift-ca" pod "9dc40c87-1067-48c8-9259-b8d7581369f5" (UID: "9dc40c87-1067-48c8-9259-b8d7581369f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.396118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.396099    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.401241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.401212    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-xks-ca" pod "9dc40c87-1067-48c8-9259-b8d7581369f5" (UID: "9dc40c87-1067-48c8-9259-b8d7581369f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:19.401241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.401222    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc40c87-1067-48c8-9259-b8d7581369f5-kube-api-access-f4gdq" pod "9dc40c87-1067-48c8-9259-b8d7581369f5" (UID: "9dc40c87-1067-48c8-9259-b8d7581369f5"). InnerVolumeSpecName "kube-api-access-f4gdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.499043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.499002    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc40c87-1067-48c8-9259-b8d7581369f5-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.499176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.499046    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gdq\" (UniqueName: \"kubernetes.io/projected/9dc40c87-1067-48c8-9259-b8d7581369f5-kube-api-access-f4gdq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.770945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:19.770810    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:20.164476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.164438    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" event={"ID":"32e94e10-b901-4c30-8080-42b33ce1a0be","Type":"ContainerStarted","Data":"688aa449fc225f2216cb462ae5cd151ce5b98865aa382b2eccdde46b4cabbe7b"}
Oct 30 19:45:20.167633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.167596    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" event={"ID":"2d051361-c3c7-4b15-8fb6-1462d7433649","Type":"ContainerStarted","Data":"532f975c652764cf76ce89adaaa78f7692d56a7ce387b25cb1303f612b993c28"}
Oct 30 19:45:20.169465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.169442    4925 generic.go:356] "Generic (PLEG): container finished" podID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerID="4dc2cbbb188054582ed49bbd744b532a3b711cbed79b22ff8777ce168c157cf8" exitCode=60
Oct 30 19:45:20.169575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.169518    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" event={"ID":"0a7417f7-d341-4c53-a5b4-7239ec97c5f3","Type":"ContainerDied","Data":"4dc2cbbb188054582ed49bbd744b532a3b711cbed79b22ff8777ce168c157cf8"}
Oct 30 19:45:20.169575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.169571    4925 scope.go:122] "RemoveContainer" containerID="d7110f91e18c484e225c8e8c80131563265bdfa09b8bf39db74c541a784ef552"
Oct 30 19:45:20.169992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.169970    4925 scope.go:122] "RemoveContainer" containerID="4dc2cbbb188054582ed49bbd744b532a3b711cbed79b22ff8777ce168c157cf8"
Oct 30 19:45:20.170382 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:20.170356    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv_odh-ai-gateway-infra(0a7417f7-d341-4c53-a5b4-7239ec97c5f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3"
Oct 30 19:45:20.171170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.171156    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"
Oct 30 19:45:20.171278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.171165    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5" event={"ID":"9dc40c87-1067-48c8-9259-b8d7581369f5","Type":"ContainerDied","Data":"05e44f905cac10628e9c0065b38accf38659a9c84130886fbe3aa9418f520a89"}
Oct 30 19:45:20.171278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.171198    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05e44f905cac10628e9c0065b38accf38659a9c84130886fbe3aa9418f520a89"
Oct 30 19:45:20.173197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.173179    4925 generic.go:356] "Generic (PLEG): container finished" podID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerID="77bf37ccd1f2ac200bdc67e770f6259bf89ff3fc45cc7f72365c3a10e7e82e71" exitCode=60
Oct 30 19:45:20.173289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.173248    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" event={"ID":"55dc9932-b47f-4181-b57f-1f021ffb67b9","Type":"ContainerDied","Data":"77bf37ccd1f2ac200bdc67e770f6259bf89ff3fc45cc7f72365c3a10e7e82e71"}
Oct 30 19:45:20.173645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.173623    4925 scope.go:122] "RemoveContainer" containerID="77bf37ccd1f2ac200bdc67e770f6259bf89ff3fc45cc7f72365c3a10e7e82e71"
Oct 30 19:45:20.173945 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:20.173924    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn_odh-ai-gateway-infra(55dc9932-b47f-4181-b57f-1f021ffb67b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9"
Oct 30 19:45:20.175385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.175365    4925 generic.go:356] "Generic (PLEG): container finished" podID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerID="d65cc905c98be0c308d3e105996e3cec2d28dd983797babd62b5af0b02988ff0" exitCode=60
Oct 30 19:45:20.175491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.175432    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" event={"ID":"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97","Type":"ContainerDied","Data":"d65cc905c98be0c308d3e105996e3cec2d28dd983797babd62b5af0b02988ff0"}
Oct 30 19:45:20.175833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.175816    4925 scope.go:122] "RemoveContainer" containerID="d65cc905c98be0c308d3e105996e3cec2d28dd983797babd62b5af0b02988ff0"
Oct 30 19:45:20.176114 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:20.176097    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh_odh-ai-gateway-infra(dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97"
Oct 30 19:45:20.177728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.177663    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" event={"ID":"61fb7c2b-8b55-4f61-b29c-1f2093448e31","Type":"ContainerStarted","Data":"3d30da1dc8f30f5e2111f08d03e9061f1c014a85f52935bd92e6304a59cf1d4e"}
Oct 30 19:45:20.187551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.187507    4925 scope.go:122] "RemoveContainer" containerID="a11f210a622102b71d90e83e8f204960085d70ff6f5509373976f45e5a08dc7f"
Oct 30 19:45:20.198780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.198749    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"]
Oct 30 19:45:20.209311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.209271    4925 scope.go:122] "RemoveContainer" containerID="0a380e33fd02e006c22d1e486eb9792d8abfe634e420d2445fe6efe32f903b7e"
Oct 30 19:45:20.219731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.219243    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"]
Oct 30 19:45:20.245215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.245186    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"]
Oct 30 19:45:20.255891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.255299    4925 scope.go:122] "RemoveContainer" containerID="d2c047998a031b8470018ece1f8005dc2ed8fea59eba9977302e772e0b71a47e"
Oct 30 19:45:20.765033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.764827    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:20.827743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.825846    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:20.832439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.832415    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:20.965197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.965167    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9455f850-77c8-474c-b383-cecddf457d92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9455f850-77c8-474c-b383-cecddf457d92-maas-api-service-ca\") pod \"9455f850-77c8-474c-b383-cecddf457d92\" (UID: \"9455f850-77c8-474c-b383-cecddf457d92\") "
Oct 30 19:45:20.965412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.965283    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75816276-0209-464f-8a2a-bd0faca897e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75816276-0209-464f-8a2a-bd0faca897e5-maas-api-service-ca\") pod \"75816276-0209-464f-8a2a-bd0faca897e5\" (UID: \"75816276-0209-464f-8a2a-bd0faca897e5\") "
Oct 30 19:45:20.965500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.965422    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9455f850-77c8-474c-b383-cecddf457d92-kube-api-access-xlrs9\" (UniqueName: \"kubernetes.io/projected/9455f850-77c8-474c-b383-cecddf457d92-kube-api-access-xlrs9\") pod \"9455f850-77c8-474c-b383-cecddf457d92\" (UID: \"9455f850-77c8-474c-b383-cecddf457d92\") "
Oct 30 19:45:20.965566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.965522    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75816276-0209-464f-8a2a-bd0faca897e5-kube-api-access-vpz5r\" (UniqueName: \"kubernetes.io/projected/75816276-0209-464f-8a2a-bd0faca897e5-kube-api-access-vpz5r\") pod \"75816276-0209-464f-8a2a-bd0faca897e5\" (UID: \"75816276-0209-464f-8a2a-bd0faca897e5\") "
Oct 30 19:45:20.965619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.965584    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9455f850-77c8-474c-b383-cecddf457d92-maas-api-service-ca" pod "9455f850-77c8-474c-b383-cecddf457d92" (UID: "9455f850-77c8-474c-b383-cecddf457d92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:20.965683 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.965636    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75816276-0209-464f-8a2a-bd0faca897e5-maas-api-service-ca" pod "75816276-0209-464f-8a2a-bd0faca897e5" (UID: "75816276-0209-464f-8a2a-bd0faca897e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:20.966928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.966897    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9455f850-77c8-474c-b383-cecddf457d92-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.967049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.966929    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75816276-0209-464f-8a2a-bd0faca897e5-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.968153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.968130    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9455f850-77c8-474c-b383-cecddf457d92-kube-api-access-xlrs9" pod "9455f850-77c8-474c-b383-cecddf457d92" (UID: "9455f850-77c8-474c-b383-cecddf457d92"). InnerVolumeSpecName "kube-api-access-xlrs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:20.968243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:20.968216    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75816276-0209-464f-8a2a-bd0faca897e5-kube-api-access-vpz5r" pod "75816276-0209-464f-8a2a-bd0faca897e5" (UID: "75816276-0209-464f-8a2a-bd0faca897e5"). InnerVolumeSpecName "kube-api-access-vpz5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.068738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.068707    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrs9\" (UniqueName: \"kubernetes.io/projected/9455f850-77c8-474c-b383-cecddf457d92-kube-api-access-xlrs9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.068738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.068737    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpz5r\" (UniqueName: \"kubernetes.io/projected/75816276-0209-464f-8a2a-bd0faca897e5-kube-api-access-vpz5r\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.192705 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.192655    4925 generic.go:356] "Generic (PLEG): container finished" podID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerID="688aa449fc225f2216cb462ae5cd151ce5b98865aa382b2eccdde46b4cabbe7b" exitCode=60
Oct 30 19:45:21.193198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.192715    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" event={"ID":"32e94e10-b901-4c30-8080-42b33ce1a0be","Type":"ContainerDied","Data":"688aa449fc225f2216cb462ae5cd151ce5b98865aa382b2eccdde46b4cabbe7b"}
Oct 30 19:45:21.193198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.192793    4925 scope.go:122] "RemoveContainer" containerID="8e40bb83729fa737bd6f6fa984e7580804962187f574e1a7b01ecc9de89451ef"
Oct 30 19:45:21.193321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.193246    4925 scope.go:122] "RemoveContainer" containerID="688aa449fc225f2216cb462ae5cd151ce5b98865aa382b2eccdde46b4cabbe7b"
Oct 30 19:45:21.193677 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:21.193648    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z_odh-ai-gateway-infra(32e94e10-b901-4c30-8080-42b33ce1a0be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be"
Oct 30 19:45:21.195221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.195194    4925 generic.go:356] "Generic (PLEG): container finished" podID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerID="532f975c652764cf76ce89adaaa78f7692d56a7ce387b25cb1303f612b993c28" exitCode=60
Oct 30 19:45:21.195354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.195272    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" event={"ID":"2d051361-c3c7-4b15-8fb6-1462d7433649","Type":"ContainerDied","Data":"532f975c652764cf76ce89adaaa78f7692d56a7ce387b25cb1303f612b993c28"}
Oct 30 19:45:21.195734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.195672    4925 scope.go:122] "RemoveContainer" containerID="532f975c652764cf76ce89adaaa78f7692d56a7ce387b25cb1303f612b993c28"
Oct 30 19:45:21.196139 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:21.196117    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs_odh-ai-gateway-infra(2d051361-c3c7-4b15-8fb6-1462d7433649)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649"
Oct 30 19:45:21.197904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.197879    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"]
Oct 30 19:45:21.202044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.202020    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" event={"ID":"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de","Type":"ContainerStarted","Data":"46e893e0e4f2a338a2be5ea2e7d50ccf7aa1710a7f8fb7c2578f457526b499c3"}
Oct 30 19:45:21.206808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.206785    4925 generic.go:356] "Generic (PLEG): container finished" podID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerID="3d30da1dc8f30f5e2111f08d03e9061f1c014a85f52935bd92e6304a59cf1d4e" exitCode=60
Oct 30 19:45:21.207058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.206906    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" event={"ID":"61fb7c2b-8b55-4f61-b29c-1f2093448e31","Type":"ContainerDied","Data":"3d30da1dc8f30f5e2111f08d03e9061f1c014a85f52935bd92e6304a59cf1d4e"}
Oct 30 19:45:21.207430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.207411    4925 scope.go:122] "RemoveContainer" containerID="3d30da1dc8f30f5e2111f08d03e9061f1c014a85f52935bd92e6304a59cf1d4e"
Oct 30 19:45:21.207798 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:21.207761    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v_odh-ai-gateway-infra(61fb7c2b-8b55-4f61-b29c-1f2093448e31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31"
Oct 30 19:45:21.209547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.209150    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"
Oct 30 19:45:21.209547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.209149    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f" event={"ID":"9455f850-77c8-474c-b383-cecddf457d92","Type":"ContainerDied","Data":"9c7d9921b3ec3574e6dd0187a03cbaa2a4103b5cb1cdd28d517513aaa4a715f8"}
Oct 30 19:45:21.211791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.211237    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc" event={"ID":"75816276-0209-464f-8a2a-bd0faca897e5","Type":"ContainerDied","Data":"e8db850ccde7bc74f02cbbf8240e4f62f4bd60b6ad5f5e09a8e8209ac1d7175a"}
Oct 30 19:45:21.211791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.211267    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"
Oct 30 19:45:21.218002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.217982    4925 scope.go:122] "RemoveContainer" containerID="d0b5600f14f02d29a377ed3fe6476f061f6392b3209b6207325c69f559c57263"
Oct 30 19:45:21.240646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.240571    4925 scope.go:122] "RemoveContainer" containerID="b7ccc04fa6753e13f7fa093c246b045fcee66a82c6b369b007a94762c9d19456"
Oct 30 19:45:21.277314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.277138    4925 scope.go:122] "RemoveContainer" containerID="124e24b8bffdf69bd92910ea4d58fa240a497fee0e68e744903d66f49a5d26fb"
Oct 30 19:45:21.279779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.279593    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" path="/var/lib/kubelet/pods/e93c80b0-0a57-4cbf-b218-06b9b5fe52c4/volumes"
Oct 30 19:45:21.280203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.280181    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"]
Oct 30 19:45:21.327814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.327794    4925 scope.go:122] "RemoveContainer" containerID="1ad1b9c60970aec7ed73e15e2e4bd55d8cb1d0735d4de7dccae06434bcd342ef"
Oct 30 19:45:21.340035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.340004    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"]
Oct 30 19:45:21.397353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.397318    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"]
Oct 30 19:45:21.406020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.405991    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ad23c7-29889825-5rdzc"]
Oct 30 19:45:21.426613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.426578    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"]
Oct 30 19:45:21.432983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.432955    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2405fb-29889825-bd97f"]
Oct 30 19:45:21.542206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.542179    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:21.553289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.553179    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:21.557534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.557515    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:21.676721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.675077    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:21.685964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.685920    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-kube-api-access-jnss7\" (UniqueName: \"kubernetes.io/projected/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-kube-api-access-jnss7\") pod \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\" (UID: \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\") "
Oct 30 19:45:21.686168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.686153    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-kube-api-access-ktg84\" (UniqueName: \"kubernetes.io/projected/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-kube-api-access-ktg84\") pod \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\" (UID: \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\") "
Oct 30 19:45:21.686280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.686193    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-maas-api-service-ca\") pod \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\" (UID: \"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97\") "
Oct 30 19:45:21.686374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.686350    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55dc9932-b47f-4181-b57f-1f021ffb67b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55dc9932-b47f-4181-b57f-1f021ffb67b9-maas-api-service-ca\") pod \"55dc9932-b47f-4181-b57f-1f021ffb67b9\" (UID: \"55dc9932-b47f-4181-b57f-1f021ffb67b9\") "
Oct 30 19:45:21.686447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.686415    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55dc9932-b47f-4181-b57f-1f021ffb67b9-kube-api-access-kvfsg\" (UniqueName: \"kubernetes.io/projected/55dc9932-b47f-4181-b57f-1f021ffb67b9-kube-api-access-kvfsg\") pod \"55dc9932-b47f-4181-b57f-1f021ffb67b9\" (UID: \"55dc9932-b47f-4181-b57f-1f021ffb67b9\") "
Oct 30 19:45:21.687084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.687060    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-maas-api-service-ca\") pod \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\" (UID: \"0a7417f7-d341-4c53-a5b4-7239ec97c5f3\") "
Oct 30 19:45:21.688997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.688974    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55dc9932-b47f-4181-b57f-1f021ffb67b9-maas-api-service-ca" pod "55dc9932-b47f-4181-b57f-1f021ffb67b9" (UID: "55dc9932-b47f-4181-b57f-1f021ffb67b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.689352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.689290    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-maas-api-service-ca" pod "0a7417f7-d341-4c53-a5b4-7239ec97c5f3" (UID: "0a7417f7-d341-4c53-a5b4-7239ec97c5f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.689670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.689645    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-maas-api-service-ca" pod "dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" (UID: "dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.693421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.693390    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-kube-api-access-ktg84" pod "0a7417f7-d341-4c53-a5b4-7239ec97c5f3" (UID: "0a7417f7-d341-4c53-a5b4-7239ec97c5f3"). InnerVolumeSpecName "kube-api-access-ktg84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.693807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.693781    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-kube-api-access-jnss7" pod "dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" (UID: "dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97"). InnerVolumeSpecName "kube-api-access-jnss7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.699706 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.696287    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55dc9932-b47f-4181-b57f-1f021ffb67b9-kube-api-access-kvfsg" pod "55dc9932-b47f-4181-b57f-1f021ffb67b9" (UID: "55dc9932-b47f-4181-b57f-1f021ffb67b9"). InnerVolumeSpecName "kube-api-access-kvfsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.789918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.789831    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.789918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.789863    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55dc9932-b47f-4181-b57f-1f021ffb67b9-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.789918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.789872    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfsg\" (UniqueName: \"kubernetes.io/projected/55dc9932-b47f-4181-b57f-1f021ffb67b9-kube-api-access-kvfsg\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.789918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.789882    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.789918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.789891    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnss7\" (UniqueName: \"kubernetes.io/projected/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97-kube-api-access-jnss7\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:21.789918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:21.789899    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktg84\" (UniqueName: \"kubernetes.io/projected/0a7417f7-d341-4c53-a5b4-7239ec97c5f3-kube-api-access-ktg84\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.127954 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:22.127861    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:45:22.228959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.228928    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"
Oct 30 19:45:22.228959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.228941    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv" event={"ID":"0a7417f7-d341-4c53-a5b4-7239ec97c5f3","Type":"ContainerDied","Data":"225cc084cc2506874a4db9eb4f7624200df1b773a7f260376f3d96d59be81d58"}
Oct 30 19:45:22.229452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.229007    4925 scope.go:122] "RemoveContainer" containerID="4dc2cbbb188054582ed49bbd744b532a3b711cbed79b22ff8777ce168c157cf8"
Oct 30 19:45:22.230438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.230422    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"
Oct 30 19:45:22.230570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.230465    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn" event={"ID":"55dc9932-b47f-4181-b57f-1f021ffb67b9","Type":"ContainerDied","Data":"96bc6f466eb08751337cd1b56258ef9c4e76d574b9989827852601fef6ff5eb4"}
Oct 30 19:45:22.233725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.233680    4925 generic.go:356] "Generic (PLEG): container finished" podID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerID="46e893e0e4f2a338a2be5ea2e7d50ccf7aa1710a7f8fb7c2578f457526b499c3" exitCode=60
Oct 30 19:45:22.233826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.233744    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" event={"ID":"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de","Type":"ContainerDied","Data":"46e893e0e4f2a338a2be5ea2e7d50ccf7aa1710a7f8fb7c2578f457526b499c3"}
Oct 30 19:45:22.234212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.234191    4925 scope.go:122] "RemoveContainer" containerID="46e893e0e4f2a338a2be5ea2e7d50ccf7aa1710a7f8fb7c2578f457526b499c3"
Oct 30 19:45:22.234581 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:22.234554    4925 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q_odh-ai-gateway-infra(3d2ddc65-5fb3-4c1b-bb55-41f8fec542de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de"
Oct 30 19:45:22.245034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.240572    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh" event={"ID":"dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97","Type":"ContainerDied","Data":"bcfbd7eb9d6b44059825502f6cdb8877622ff33b2352ab8c0f0ab9c134dac0eb"}
Oct 30 19:45:22.245034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.240717    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"
Oct 30 19:45:22.251217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.251100    4925 scope.go:122] "RemoveContainer" containerID="77bf37ccd1f2ac200bdc67e770f6259bf89ff3fc45cc7f72365c3a10e7e82e71"
Oct 30 19:45:22.276740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.276590    4925 scope.go:122] "RemoveContainer" containerID="d2c047998a031b8470018ece1f8005dc2ed8fea59eba9977302e772e0b71a47e"
Oct 30 19:45:22.322950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.322652    4925 scope.go:122] "RemoveContainer" containerID="d65cc905c98be0c308d3e105996e3cec2d28dd983797babd62b5af0b02988ff0"
Oct 30 19:45:22.344177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.344086    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"]
Oct 30 19:45:22.352653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.352610    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"]
Oct 30 19:45:22.362610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.362078    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e7123-29889825-mxclv"]
Oct 30 19:45:22.384487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.384422    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"]
Oct 30 19:45:22.388592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.388565    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1b1531-29889825-c88xn"]
Oct 30 19:45:22.415727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.407666    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"]
Oct 30 19:45:22.463254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.445289    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17e904-29889825-glkgh"]
Oct 30 19:45:22.521463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.520972    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:22.527873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.527569    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:22.615985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.615468    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d051361-c3c7-4b15-8fb6-1462d7433649-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d051361-c3c7-4b15-8fb6-1462d7433649-maas-api-service-ca\") pod \"2d051361-c3c7-4b15-8fb6-1462d7433649\" (UID: \"2d051361-c3c7-4b15-8fb6-1462d7433649\") "
Oct 30 19:45:22.615985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.615602    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d051361-c3c7-4b15-8fb6-1462d7433649-kube-api-access-x4xq2\" (UniqueName: \"kubernetes.io/projected/2d051361-c3c7-4b15-8fb6-1462d7433649-kube-api-access-x4xq2\") pod \"2d051361-c3c7-4b15-8fb6-1462d7433649\" (UID: \"2d051361-c3c7-4b15-8fb6-1462d7433649\") "
Oct 30 19:45:22.615985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.615905    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d051361-c3c7-4b15-8fb6-1462d7433649-maas-api-service-ca" pod "2d051361-c3c7-4b15-8fb6-1462d7433649" (UID: "2d051361-c3c7-4b15-8fb6-1462d7433649"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:22.616531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.616505    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d051361-c3c7-4b15-8fb6-1462d7433649-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.618477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.618455    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:22.618707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.618661    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d051361-c3c7-4b15-8fb6-1462d7433649-kube-api-access-x4xq2" pod "2d051361-c3c7-4b15-8fb6-1462d7433649" (UID: "2d051361-c3c7-4b15-8fb6-1462d7433649"). InnerVolumeSpecName "kube-api-access-x4xq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:22.623893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.623863    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:22.717742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.717674    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e94e10-b901-4c30-8080-42b33ce1a0be-kube-api-access-n6wq8\" (UniqueName: \"kubernetes.io/projected/32e94e10-b901-4c30-8080-42b33ce1a0be-kube-api-access-n6wq8\") pod \"32e94e10-b901-4c30-8080-42b33ce1a0be\" (UID: \"32e94e10-b901-4c30-8080-42b33ce1a0be\") "
Oct 30 19:45:22.717898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.717775    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61fb7c2b-8b55-4f61-b29c-1f2093448e31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fb7c2b-8b55-4f61-b29c-1f2093448e31-maas-api-service-ca\") pod \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\" (UID: \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\") "
Oct 30 19:45:22.717898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.717824    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fb7c2b-8b55-4f61-b29c-1f2093448e31-kube-api-access-kqljj\" (UniqueName: \"kubernetes.io/projected/61fb7c2b-8b55-4f61-b29c-1f2093448e31-kube-api-access-kqljj\") pod \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\" (UID: \"61fb7c2b-8b55-4f61-b29c-1f2093448e31\") "
Oct 30 19:45:22.718014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.717922    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e94e10-b901-4c30-8080-42b33ce1a0be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e94e10-b901-4c30-8080-42b33ce1a0be-maas-api-service-ca\") pod \"32e94e10-b901-4c30-8080-42b33ce1a0be\" (UID: \"32e94e10-b901-4c30-8080-42b33ce1a0be\") "
Oct 30 19:45:22.718200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.718128    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fb7c2b-8b55-4f61-b29c-1f2093448e31-maas-api-service-ca" pod "61fb7c2b-8b55-4f61-b29c-1f2093448e31" (UID: "61fb7c2b-8b55-4f61-b29c-1f2093448e31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:22.718322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.718288    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e94e10-b901-4c30-8080-42b33ce1a0be-maas-api-service-ca" pod "32e94e10-b901-4c30-8080-42b33ce1a0be" (UID: "32e94e10-b901-4c30-8080-42b33ce1a0be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:22.719005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.718984    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4xq2\" (UniqueName: \"kubernetes.io/projected/2d051361-c3c7-4b15-8fb6-1462d7433649-kube-api-access-x4xq2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.719073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.719013    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fb7c2b-8b55-4f61-b29c-1f2093448e31-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.719073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.719029    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e94e10-b901-4c30-8080-42b33ce1a0be-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.720165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.720144    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fb7c2b-8b55-4f61-b29c-1f2093448e31-kube-api-access-kqljj" pod "61fb7c2b-8b55-4f61-b29c-1f2093448e31" (UID: "61fb7c2b-8b55-4f61-b29c-1f2093448e31"). InnerVolumeSpecName "kube-api-access-kqljj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:22.720225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.720166    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e94e10-b901-4c30-8080-42b33ce1a0be-kube-api-access-n6wq8" pod "32e94e10-b901-4c30-8080-42b33ce1a0be" (UID: "32e94e10-b901-4c30-8080-42b33ce1a0be"). InnerVolumeSpecName "kube-api-access-n6wq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:22.820547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.820518    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6wq8\" (UniqueName: \"kubernetes.io/projected/32e94e10-b901-4c30-8080-42b33ce1a0be-kube-api-access-n6wq8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:22.820547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:22.820542    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqljj\" (UniqueName: \"kubernetes.io/projected/61fb7c2b-8b55-4f61-b29c-1f2093448e31-kube-api-access-kqljj\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:23.015878 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:23.015790    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:23.270300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.270198    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:23.293050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.293017    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" path="/var/lib/kubelet/pods/0a7417f7-d341-4c53-a5b4-7239ec97c5f3/volumes"
Oct 30 19:45:23.294168 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.294146    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" path="/var/lib/kubelet/pods/55dc9932-b47f-4181-b57f-1f021ffb67b9/volumes"
Oct 30 19:45:23.295139 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.295110    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75816276-0209-464f-8a2a-bd0faca897e5" path="/var/lib/kubelet/pods/75816276-0209-464f-8a2a-bd0faca897e5/volumes"
Oct 30 19:45:23.296142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.296121    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9455f850-77c8-474c-b383-cecddf457d92" path="/var/lib/kubelet/pods/9455f850-77c8-474c-b383-cecddf457d92/volumes"
Oct 30 19:45:23.296831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.296676    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" path="/var/lib/kubelet/pods/dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97/volumes"
Oct 30 19:45:23.305895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.305867    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v" event={"ID":"61fb7c2b-8b55-4f61-b29c-1f2093448e31","Type":"ContainerDied","Data":"8b179ef6589eaf9b63b30caae7a4af4a15433b5ea8778a0c967bfe3b6cc91940"}
Oct 30 19:45:23.306001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.305923    4925 scope.go:122] "RemoveContainer" containerID="3d30da1dc8f30f5e2111f08d03e9061f1c014a85f52935bd92e6304a59cf1d4e"
Oct 30 19:45:23.306073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.306028    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"
Oct 30 19:45:23.311771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.311222    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z" event={"ID":"32e94e10-b901-4c30-8080-42b33ce1a0be","Type":"ContainerDied","Data":"d25621c8d44e56b3eb383eaa6da5c3ff288ae0ce405649bf64c78dcbade03929"}
Oct 30 19:45:23.311771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.311249    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"
Oct 30 19:45:23.316401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.316368    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs" event={"ID":"2d051361-c3c7-4b15-8fb6-1462d7433649","Type":"ContainerDied","Data":"e9ea8439934e3c1230093dd7dff0d2d74e740f3b01c21daf4f3420bb733d74fd"}
Oct 30 19:45:23.316507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.316410    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"
Oct 30 19:45:23.327461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.327439    4925 scope.go:122] "RemoveContainer" containerID="688aa449fc225f2216cb462ae5cd151ce5b98865aa382b2eccdde46b4cabbe7b"
Oct 30 19:45:23.341343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.341324    4925 scope.go:122] "RemoveContainer" containerID="532f975c652764cf76ce89adaaa78f7692d56a7ce387b25cb1303f612b993c28"
Oct 30 19:45:23.368359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.368332    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"]
Oct 30 19:45:23.373864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.373837    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cfa4bb-29889825-vdx5v"]
Oct 30 19:45:23.378148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.378120    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"]
Oct 30 19:45:23.383228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.383200    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-fd7c77-29889825-zlr5z"]
Oct 30 19:45:23.395608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.395470    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"]
Oct 30 19:45:23.408318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.408287    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1a844e-29889825-f7szs"]
Oct 30 19:45:23.502377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.502344    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:23.557641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.557556    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-maas-api-service-ca\") pod \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\" (UID: \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\") "
Oct 30 19:45:23.557832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.557644    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-kube-api-access-tvn7v\" (UniqueName: \"kubernetes.io/projected/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-kube-api-access-tvn7v\") pod \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\" (UID: \"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de\") "
Oct 30 19:45:23.558010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.557988    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-maas-api-service-ca" pod "3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" (UID: "3d2ddc65-5fb3-4c1b-bb55-41f8fec542de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:23.558445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.558425    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-maas-api-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:23.559987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.559965    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-kube-api-access-tvn7v" pod "3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" (UID: "3d2ddc65-5fb3-4c1b-bb55-41f8fec542de"). InnerVolumeSpecName "kube-api-access-tvn7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:23.660113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.660075    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvn7v\" (UniqueName: \"kubernetes.io/projected/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de-kube-api-access-tvn7v\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:23.829393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.829299    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:23.860548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.860514    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"]
Oct 30 19:45:23.861310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861289    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9455f850-77c8-474c-b383-cecddf457d92" containerName="cleanup"
Oct 30 19:45:23.861378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861313    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9455f850-77c8-474c-b383-cecddf457d92" containerName="cleanup"
Oct 30 19:45:23.861378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861337    4925 cpu_manager.go:414] "containerMap: removing container" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerName="cleanup"
Oct 30 19:45:23.861378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861346    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerName="cleanup"
Oct 30 19:45:23.861378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861367    4925 cpu_manager.go:414] "containerMap: removing container" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerName="cleanup"
Oct 30 19:45:23.861378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861375    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861386    4925 cpu_manager.go:414] "containerMap: removing container" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861394    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861405    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861412    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861420    4925 cpu_manager.go:414] "containerMap: removing container" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861428    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861438    4925 cpu_manager.go:414] "containerMap: removing container" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861445    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861478    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861485    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861497    4925 cpu_manager.go:414] "containerMap: removing container" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861503    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861509    4925 cpu_manager.go:414] "containerMap: removing container" podUID="200f584f-edfd-4c92-8b9d-7251fbdc1b0e" containerName="revoke-keys"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861513    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200f584f-edfd-4c92-8b9d-7251fbdc1b0e" containerName="revoke-keys"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861520    4925 cpu_manager.go:414] "containerMap: removing container" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861526    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861535    4925 cpu_manager.go:414] "containerMap: removing container" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861542    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861553    4925 cpu_manager.go:414] "containerMap: removing container" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861561    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861577    4925 cpu_manager.go:414] "containerMap: removing container" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861583    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861593    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861600    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerName="cleanup"
Oct 30 19:45:23.861599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861609    4925 cpu_manager.go:414] "containerMap: removing container" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861614    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861622    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861626    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861640    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9455f850-77c8-474c-b383-cecddf457d92" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861644    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9455f850-77c8-474c-b383-cecddf457d92" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861650    4925 cpu_manager.go:414] "containerMap: removing container" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861654    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861660    4925 cpu_manager.go:414] "containerMap: removing container" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861664    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861674    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9455f850-77c8-474c-b383-cecddf457d92" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861678    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9455f850-77c8-474c-b383-cecddf457d92" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861700    4925 cpu_manager.go:414] "containerMap: removing container" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861705    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861711    4925 cpu_manager.go:414] "containerMap: removing container" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861716    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861723    4925 cpu_manager.go:414] "containerMap: removing container" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861727    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861736    4925 cpu_manager.go:414] "containerMap: removing container" podUID="75816276-0209-464f-8a2a-bd0faca897e5" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861740    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75816276-0209-464f-8a2a-bd0faca897e5" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861752    4925 cpu_manager.go:414] "containerMap: removing container" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861756    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861763    4925 cpu_manager.go:414] "containerMap: removing container" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861767    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861775    4925 cpu_manager.go:414] "containerMap: removing container" podUID="75816276-0209-464f-8a2a-bd0faca897e5" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861779    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75816276-0209-464f-8a2a-bd0faca897e5" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861789    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9dc40c87-1067-48c8-9259-b8d7581369f5" containerName="revoke-keys"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861793    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc40c87-1067-48c8-9259-b8d7581369f5" containerName="revoke-keys"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861801    4925 cpu_manager.go:414] "containerMap: removing container" podUID="75816276-0209-464f-8a2a-bd0faca897e5" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861805    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75816276-0209-464f-8a2a-bd0faca897e5" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861815    4925 cpu_manager.go:414] "containerMap: removing container" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861820    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861959    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861966    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861973    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9455f850-77c8-474c-b383-cecddf457d92" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861980    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861984    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861991    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.861997    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="75816276-0209-464f-8a2a-bd0faca897e5" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862004    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862009    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862015    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="75816276-0209-464f-8a2a-bd0faca897e5" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862020    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862024    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc40c87-1067-48c8-9259-b8d7581369f5" containerName="revoke-keys"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862030    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="75816276-0209-464f-8a2a-bd0faca897e5" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862038    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862045    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9455f850-77c8-474c-b383-cecddf457d92" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862051    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862057    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862063    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862068    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7417f7-d341-4c53-a5b4-7239ec97c5f3" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862076    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862081    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="200f584f-edfd-4c92-8b9d-7251fbdc1b0e" containerName="revoke-keys"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862087    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862092    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862097    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7febe178-dee4-4add-9355-5da5c8dfc1ee" containerName="istio-proxy"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862103    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862108    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862112    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" containerName="cleanup"
Oct 30 19:45:23.862422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.862119    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerName="cleanup"
Oct 30 19:45:23.866130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.866114    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:23.873091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.873055    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"]
Oct 30 19:45:23.875644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.875619    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7aa217\""
Oct 30 19:45:23.875963 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.875940    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7aa217-dockercfg-bzh6b\""
Oct 30 19:45:23.887800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.887770    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"]
Oct 30 19:45:23.888986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.888960    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerName="cleanup"
Oct 30 19:45:23.888986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.888986    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" containerName="cleanup"
Oct 30 19:45:23.889205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889036    4925 cpu_manager.go:414] "containerMap: removing container" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerName="cleanup"
Oct 30 19:45:23.889205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889045    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerName="cleanup"
Oct 30 19:45:23.889205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889114    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerName="cleanup"
Oct 30 19:45:23.889205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889122    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerName="cleanup"
Oct 30 19:45:23.889438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889285    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93c80b0-0a57-4cbf-b218-06b9b5fe52c4" containerName="cleanup"
Oct 30 19:45:23.889438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889305    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9455f850-77c8-474c-b383-cecddf457d92" containerName="cleanup"
Oct 30 19:45:23.889438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889319    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" containerName="cleanup"
Oct 30 19:45:23.889438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889332    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" containerName="cleanup"
Oct 30 19:45:23.889438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889351    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8510bc-b8bc-455a-8c09-7b3a7ed2dc97" containerName="cleanup"
Oct 30 19:45:23.889438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.889359    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="55dc9932-b47f-4181-b57f-1f021ffb67b9" containerName="cleanup"
Oct 30 19:45:23.894032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.894012    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:23.904722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:23.904675    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"]
Oct 30 19:45:24.031709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.030292    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:24.099766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.096795    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vjt8\" (UniqueName: \"kubernetes.io/projected/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-kube-api-access-9vjt8\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz\" (UID: \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:24.099766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.097114    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b403950e-962e-4a8c-99e9-5baee57b2d55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk\" (UID: \"b403950e-962e-4a8c-99e9-5baee57b2d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:24.099766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.097161    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz\" (UID: \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:24.099766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.097271    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kmh\" (UniqueName: \"kubernetes.io/projected/b403950e-962e-4a8c-99e9-5baee57b2d55-kube-api-access-m7kmh\") pod \"payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk\" (UID: \"b403950e-962e-4a8c-99e9-5baee57b2d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:24.205713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.200825    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b403950e-962e-4a8c-99e9-5baee57b2d55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk\" (UID: \"b403950e-962e-4a8c-99e9-5baee57b2d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:24.205713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.200888    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz\" (UID: \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:24.205713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.201011    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kmh\" (UniqueName: \"kubernetes.io/projected/b403950e-962e-4a8c-99e9-5baee57b2d55-kube-api-access-m7kmh\") pod \"payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk\" (UID: \"b403950e-962e-4a8c-99e9-5baee57b2d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:24.205713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.201068    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vjt8\" (UniqueName: \"kubernetes.io/projected/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-kube-api-access-9vjt8\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz\" (UID: \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:24.205713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.202354    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b403950e-962e-4a8c-99e9-5baee57b2d55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk\" (UID: \"b403950e-962e-4a8c-99e9-5baee57b2d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:24.245319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.245231    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:24.271768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.271734    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz\" (UID: \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:24.310959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.308625    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vjt8\" (UniqueName: \"kubernetes.io/projected/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-kube-api-access-9vjt8\") pod \"payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz\" (UID: \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:24.311742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.311720    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kmh\" (UniqueName: \"kubernetes.io/projected/b403950e-962e-4a8c-99e9-5baee57b2d55-kube-api-access-m7kmh\") pod \"payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk\" (UID: \"b403950e-962e-4a8c-99e9-5baee57b2d55\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:24.316674 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:24.316645    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:24.340333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.340308    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"
Oct 30 19:45:24.340512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.340348    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q" event={"ID":"3d2ddc65-5fb3-4c1b-bb55-41f8fec542de","Type":"ContainerDied","Data":"3cf66983da4ed1de392751b8cec081643a8cc4bae6a606f13d5bda8c2c4e6488"}
Oct 30 19:45:24.340512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.340417    4925 scope.go:122] "RemoveContainer" containerID="46e893e0e4f2a338a2be5ea2e7d50ccf7aa1710a7f8fb7c2578f457526b499c3"
Oct 30 19:45:24.500716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.498332    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:24.607919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.607792    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:24.617858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.615872    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:24.825785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.795148    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:24.825785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.819752    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"]
Oct 30 19:45:24.831577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.831552    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39b867-29889825-v8r5q"]
Oct 30 19:45:24.850143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.850100    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"]
Oct 30 19:45:24.966828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:24.966790    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:25.210421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.210241    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:25.427903 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:25.427632    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c33c16f_f0f0_43cf_9172_ae33dc0c9125.slice/crio-0de53512878f4735d55ce31d07528e1d6ca4ee797f75cc05dd8adecad49d636d WatchSource:0}: Error finding container 0de53512878f4735d55ce31d07528e1d6ca4ee797f75cc05dd8adecad49d636d: Status 404 returned error can't find the container with id 0de53512878f4735d55ce31d07528e1d6ca4ee797f75cc05dd8adecad49d636d
Oct 30 19:45:25.443762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.443422    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d051361-c3c7-4b15-8fb6-1462d7433649" path="/var/lib/kubelet/pods/2d051361-c3c7-4b15-8fb6-1462d7433649/volumes"
Oct 30 19:45:25.444159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.444093    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e94e10-b901-4c30-8080-42b33ce1a0be" path="/var/lib/kubelet/pods/32e94e10-b901-4c30-8080-42b33ce1a0be/volumes"
Oct 30 19:45:25.446526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.444764    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2ddc65-5fb3-4c1b-bb55-41f8fec542de" path="/var/lib/kubelet/pods/3d2ddc65-5fb3-4c1b-bb55-41f8fec542de/volumes"
Oct 30 19:45:25.446526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.445808    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fb7c2b-8b55-4f61-b29c-1f2093448e31" path="/var/lib/kubelet/pods/61fb7c2b-8b55-4f61-b29c-1f2093448e31/volumes"
Oct 30 19:45:25.446526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.446309    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"]
Oct 30 19:45:25.450590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.450569    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"]
Oct 30 19:45:25.536626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.536595    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz" event={"ID":"2c33c16f-f0f0-43cf-9172-ae33dc0c9125","Type":"ContainerStarted","Data":"0de53512878f4735d55ce31d07528e1d6ca4ee797f75cc05dd8adecad49d636d"}
Oct 30 19:45:25.538232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.538202    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk" event={"ID":"b403950e-962e-4a8c-99e9-5baee57b2d55","Type":"ContainerStarted","Data":"2f1421d2d39ffb09c1654096a2f7625873befa3c6a21254e29df46b566dfd22f"}
Oct 30 19:45:25.821488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:25.821404    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:26.010031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.009990    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:26.132852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.132756    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"]
Oct 30 19:45:26.155724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.147231    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20b3c92-hr7f6"]
Oct 30 19:45:26.613387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.613266    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:26.619033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.617795    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz" event={"ID":"2c33c16f-f0f0-43cf-9172-ae33dc0c9125","Type":"ContainerStarted","Data":"24319253c6079647e5ef86b89667a127d86615275cfcfff9838b52c42da594dd"}
Oct 30 19:45:26.619033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.618965    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:26.621712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.621677    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk" event={"ID":"b403950e-962e-4a8c-99e9-5baee57b2d55","Type":"ContainerStarted","Data":"9ce64f37774b440ef826c849512d5e2c2cef86923e67a49bbfad157e21b6fa1a"}
Oct 30 19:45:26.622819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.622802    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:26.644134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.644087    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz" podStartSLOduration=3.644070522 podStartE2EDuration="3.644070522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:26.63958589 +0000 UTC m=+2361.542598926" watchObservedRunningTime="2026-10-30 19:45:26.644070522 +0000 UTC m=+2361.547083559"
Oct 30 19:45:26.658072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:26.658032    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk" podStartSLOduration=3.658019023 podStartE2EDuration="3.658019023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:26.656518042 +0000 UTC m=+2361.559531080" watchObservedRunningTime="2026-10-30 19:45:26.658019023 +0000 UTC m=+2361.561032059"
Oct 30 19:45:27.211016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:27.210977    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:27.267116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:27.267081    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd24c0cb-dd4d-4213-a781-d358c1fbb0b0" path="/var/lib/kubelet/pods/bd24c0cb-dd4d-4213-a781-d358c1fbb0b0/volumes"
Oct 30 19:45:27.410276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:27.410242    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:27.648771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:27.640487    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:27.692525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:27.690315    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"]
Oct 30 19:45:27.704642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:27.704601    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-618d93-btvx8"]
Oct 30 19:45:27.845838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:27.845795    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:28.031474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:28.031376    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:28.214668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:28.214634    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:28.414124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:28.414092    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:28.880783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:28.879237    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:29.123264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:29.123229    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:29.346713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:29.343538    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200f584f-edfd-4c92-8b9d-7251fbdc1b0e" path="/var/lib/kubelet/pods/200f584f-edfd-4c92-8b9d-7251fbdc1b0e/volumes"
Oct 30 19:45:29.586891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:29.586861    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:29.768438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:29.768395    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:29.860724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:29.856658    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s" podUID="173abd12-7db2-472b-bbe3-7b5d2310f12c" containerName="payload-pre-processing" containerID="cri-o://f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34" gracePeriod=25
Oct 30 19:45:29.932981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:29.932949    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:30.213240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.213201    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:45:30.303946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.303854    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/173abd12-7db2-472b-bbe3-7b5d2310f12c-kube-api-access-rvdc9\" (UniqueName: \"kubernetes.io/projected/173abd12-7db2-472b-bbe3-7b5d2310f12c-kube-api-access-rvdc9\") pod \"173abd12-7db2-472b-bbe3-7b5d2310f12c\" (UID: \"173abd12-7db2-472b-bbe3-7b5d2310f12c\") "
Oct 30 19:45:30.304106 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.304087    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/173abd12-7db2-472b-bbe3-7b5d2310f12c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173abd12-7db2-472b-bbe3-7b5d2310f12c-plugins-config-volume\") pod \"173abd12-7db2-472b-bbe3-7b5d2310f12c\" (UID: \"173abd12-7db2-472b-bbe3-7b5d2310f12c\") "
Oct 30 19:45:30.304537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.304448    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173abd12-7db2-472b-bbe3-7b5d2310f12c-plugins-config-volume" pod "173abd12-7db2-472b-bbe3-7b5d2310f12c" (UID: "173abd12-7db2-472b-bbe3-7b5d2310f12c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:30.305348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.305327    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173abd12-7db2-472b-bbe3-7b5d2310f12c-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:30.306352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.306333    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173abd12-7db2-472b-bbe3-7b5d2310f12c-kube-api-access-rvdc9" pod "173abd12-7db2-472b-bbe3-7b5d2310f12c" (UID: "173abd12-7db2-472b-bbe3-7b5d2310f12c"). InnerVolumeSpecName "kube-api-access-rvdc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:30.406883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.406851    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvdc9\" (UniqueName: \"kubernetes.io/projected/173abd12-7db2-472b-bbe3-7b5d2310f12c-kube-api-access-rvdc9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:30.589016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.588572    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:30.745782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.745670    4925 generic.go:356] "Generic (PLEG): container finished" podID="173abd12-7db2-472b-bbe3-7b5d2310f12c" containerID="f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34" exitCode=0
Oct 30 19:45:30.745782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.745752    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"
Oct 30 19:45:30.745782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.745755    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s" event={"ID":"173abd12-7db2-472b-bbe3-7b5d2310f12c","Type":"ContainerDied","Data":"f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34"}
Oct 30 19:45:30.746092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.745808    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s" event={"ID":"173abd12-7db2-472b-bbe3-7b5d2310f12c","Type":"ContainerDied","Data":"d653a7f80ffe4b641cc866e5c8de177dda4be344040f798ff1a9708d537d1ee8"}
Oct 30 19:45:30.746092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.745838    4925 scope.go:122] "RemoveContainer" containerID="f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34"
Oct 30 19:45:30.758029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.758010    4925 scope.go:122] "RemoveContainer" containerID="f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34"
Oct 30 19:45:30.758300 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:30.758279    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34\": container with ID starting with f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34 not found: ID does not exist" containerID="f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34"
Oct 30 19:45:30.758368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.758308    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34"} err="failed to get container status \"f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34\": rpc error: code = NotFound desc = could not find container \"f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34\": container with ID starting with f8c5bfde42f716aae0887d66d373e19ae6e534c4147f93c3090800b77a068f34 not found: ID does not exist"
Oct 30 19:45:30.770036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.769936    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"]
Oct 30 19:45:30.774314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:30.774287    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a844e-84f6d75f55-k5s8s"]
Oct 30 19:45:31.265627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:31.265586    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="173abd12-7db2-472b-bbe3-7b5d2310f12c" path="/var/lib/kubelet/pods/173abd12-7db2-472b-bbe3-7b5d2310f12c/volumes"
Oct 30 19:45:31.497086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:31.497047    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:32.183714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:32.182702    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:33.030896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:33.030859    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:33.100590 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:33.100554    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:33.765712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:33.760046    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:34.383246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:34.383210    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:34.561712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:34.559201    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:34.728377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:34.728336    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:34.766724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:34.731760    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"]
Oct 30 19:45:34.780038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:34.779998    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5613a-n29t5"]
Oct 30 19:45:35.310645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:35.310602    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:35.347842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:35.346663    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc40c87-1067-48c8-9259-b8d7581369f5" path="/var/lib/kubelet/pods/9dc40c87-1067-48c8-9259-b8d7581369f5/volumes"
Oct 30 19:45:35.478095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:35.478046    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:35.971547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:35.966423    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:36.164727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.161256    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:36.313982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.313887    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"]
Oct 30 19:45:36.314853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.314138    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy" containerID="cri-o://36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c" gracePeriod=30
Oct 30 19:45:36.322812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.322780    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"]
Oct 30 19:45:36.323070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.323024    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy" containerID="cri-o://e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5" gracePeriod=30
Oct 30 19:45:36.330949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.330921    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"]
Oct 30 19:45:36.331146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.331124    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy" containerID="cri-o://634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db" gracePeriod=30
Oct 30 19:45:36.468850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.467259    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:36.929828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.929794    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"]
Oct 30 19:45:36.930513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.930497    4925 cpu_manager.go:414] "containerMap: removing container" podUID="173abd12-7db2-472b-bbe3-7b5d2310f12c" containerName="payload-pre-processing"
Oct 30 19:45:36.930558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.930514    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="173abd12-7db2-472b-bbe3-7b5d2310f12c" containerName="payload-pre-processing"
Oct 30 19:45:36.930646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.930637    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="173abd12-7db2-472b-bbe3-7b5d2310f12c" containerName="payload-pre-processing"
Oct 30 19:45:36.935696 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.935668    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:36.945449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.945420    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"]
Oct 30 19:45:36.947187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.947163    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:45:36.947311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:36.947240    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:37.088285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.088251    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.088543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.088518    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbf4\" (UniqueName: \"kubernetes.io/projected/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-kube-api-access-mrbf4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.088628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.088613    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.211291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.211151    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbf4\" (UniqueName: \"kubernetes.io/projected/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-kube-api-access-mrbf4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.211483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.211318    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.211483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.211433    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.223761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.219755    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.223761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.222884    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.228230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.227938    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:37.231199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.231159    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbf4\" (UniqueName: \"kubernetes.io/projected/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-kube-api-access-mrbf4\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.255344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.255312    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:37.543728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:37.543659    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"]
Oct 30 19:45:37.544755 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:37.544727    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f94ec1_79c0_4ee5_9a54_5f99971c1b5f.slice/crio-40459a3a1a10d4e27473ef324c5838e3c6c98ec92daf89bb2ea60beb690f8163 WatchSource:0}: Error finding container 40459a3a1a10d4e27473ef324c5838e3c6c98ec92daf89bb2ea60beb690f8163: Status 404 returned error can't find the container with id 40459a3a1a10d4e27473ef324c5838e3c6c98ec92daf89bb2ea60beb690f8163
Oct 30 19:45:38.061869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:38.061717    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:38.379418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:38.379329    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm" event={"ID":"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f","Type":"ContainerStarted","Data":"d713baf9f082552aa43871b680c6142db05bbfa792db7cc117a7a46044bfc14b"}
Oct 30 19:45:38.379418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:38.379393    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm" event={"ID":"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f","Type":"ContainerStarted","Data":"40459a3a1a10d4e27473ef324c5838e3c6c98ec92daf89bb2ea60beb690f8163"}
Oct 30 19:45:38.396990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:38.396946    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm" podStartSLOduration=2.396930139 podStartE2EDuration="2.396930139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:38.395129506 +0000 UTC m=+2373.298142549" watchObservedRunningTime="2026-10-30 19:45:38.396930139 +0000 UTC m=+2373.299943176"
Oct 30 19:45:38.731674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:38.731633    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:45:38.780722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:38.771496    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:38.788942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:38.788910    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:45:39.346126 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:39.346094    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:39.389744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:39.389712    4925 generic.go:356] "Generic (PLEG): container finished" podID="d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f" containerID="d713baf9f082552aa43871b680c6142db05bbfa792db7cc117a7a46044bfc14b" exitCode=0
Oct 30 19:45:39.389915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:39.389756    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm" event={"ID":"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f","Type":"ContainerDied","Data":"d713baf9f082552aa43871b680c6142db05bbfa792db7cc117a7a46044bfc14b"}
Oct 30 19:45:39.567573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:39.565100    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:40.027245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.027205    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:40.205718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.204649    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:40.626353 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.626329    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:40.702775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.702034    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:40.733970 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.733933    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-xks-ca\") pod \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") "
Oct 30 19:45:40.734178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.734019    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-kube-api-access-mrbf4\" (UniqueName: \"kubernetes.io/projected/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-kube-api-access-mrbf4\") pod \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") "
Oct 30 19:45:40.734449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.734429    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-openshift-ca\") pod \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\" (UID: \"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f\") "
Oct 30 19:45:40.734909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.734876    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-openshift-ca" pod "d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f" (UID: "d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:40.735506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.735479    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.737178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.737150    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-xks-ca" pod "d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f" (UID: "d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:40.737178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.737162    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-kube-api-access-mrbf4" pod "d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f" (UID: "d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f"). InnerVolumeSpecName "kube-api-access-mrbf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:40.837636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.837602    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.837636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:40.837628    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrbf4\" (UniqueName: \"kubernetes.io/projected/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f-kube-api-access-mrbf4\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.454681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.452905    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:41.486430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.486310    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"
Oct 30 19:45:41.486430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.486314    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm" event={"ID":"d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f","Type":"ContainerDied","Data":"40459a3a1a10d4e27473ef324c5838e3c6c98ec92daf89bb2ea60beb690f8163"}
Oct 30 19:45:41.486430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.486384    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40459a3a1a10d4e27473ef324c5838e3c6c98ec92daf89bb2ea60beb690f8163"
Oct 30 19:45:41.780750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.780726    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:45:41.809911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.809840    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:45:41.811901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.811881    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:45:41.874209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874175    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-envoy\") pod \"4212dbf9-58f5-4408-a202-d25f052ded6b\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") "
Oct 30 19:45:41.874461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874234    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-data\") pod \"f00f570b-588f-4e4d-839f-423a7fa75cdf\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") "
Oct 30 19:45:41.874461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874278    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-socket\") pod \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") "
Oct 30 19:45:41.874461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874327    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-certs\") pod \"f00f570b-588f-4e4d-839f-423a7fa75cdf\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") "
Oct 30 19:45:41.874461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874393    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-envoy\") pod \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") "
Oct 30 19:45:41.874461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874427    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-podinfo\") pod \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") "
Oct 30 19:45:41.874461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874459    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-kube-api-access-j647l\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-kube-api-access-j647l\") pod \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") "
Oct 30 19:45:41.874865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874537    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-certs\") pod \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") "
Oct 30 19:45:41.874865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874571    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-credential-socket\") pod \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") "
Oct 30 19:45:41.874865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874612    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00f570b-588f-4e4d-839f-423a7fa75cdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f570b-588f-4e4d-839f-423a7fa75cdf-istiod-ca-cert\") pod \"f00f570b-588f-4e4d-839f-423a7fa75cdf\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") "
Oct 30 19:45:41.874865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874670    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-podinfo\") pod \"f00f570b-588f-4e4d-839f-423a7fa75cdf\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") "
Oct 30 19:45:41.874865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874806    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4212dbf9-58f5-4408-a202-d25f052ded6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212dbf9-58f5-4408-a202-d25f052ded6b-istiod-ca-cert\") pod \"4212dbf9-58f5-4408-a202-d25f052ded6b\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") "
Oct 30 19:45:41.874865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874847    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-socket\") pod \"4212dbf9-58f5-4408-a202-d25f052ded6b\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") "
Oct 30 19:45:41.875109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874906    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-data\") pod \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") "
Oct 30 19:45:41.875109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874916    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-socket" pod "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" (UID: "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.875109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874943    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-kube-api-access-wdrmx\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-kube-api-access-wdrmx\") pod \"4212dbf9-58f5-4408-a202-d25f052ded6b\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") "
Oct 30 19:45:41.875109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.874974    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-credential-socket\") pod \"4212dbf9-58f5-4408-a202-d25f052ded6b\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") "
Oct 30 19:45:41.875109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875042    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-certs\") pod \"4212dbf9-58f5-4408-a202-d25f052ded6b\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") "
Oct 30 19:45:41.875109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875082    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-data\") pod \"4212dbf9-58f5-4408-a202-d25f052ded6b\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") "
Oct 30 19:45:41.875631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875186    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-certs" pod "f00f570b-588f-4e4d-839f-423a7fa75cdf" (UID: "f00f570b-588f-4e4d-839f-423a7fa75cdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.875631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875274    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-envoy\") pod \"f00f570b-588f-4e4d-839f-423a7fa75cdf\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") "
Oct 30 19:45:41.875631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875385    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-token\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-token\") pod \"4212dbf9-58f5-4408-a202-d25f052ded6b\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") "
Oct 30 19:45:41.875631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875419    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-credential-socket" pod "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" (UID: "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.875631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875426    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istiod-ca-cert\") pod \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") "
Oct 30 19:45:41.875631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875474    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-socket\") pod \"f00f570b-588f-4e4d-839f-423a7fa75cdf\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") "
Oct 30 19:45:41.875631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875574    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-credential-socket\") pod \"f00f570b-588f-4e4d-839f-423a7fa75cdf\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") "
Oct 30 19:45:41.875631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875592    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-certs" pod "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" (UID: "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.875631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875621    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-token\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-token\") pod \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\" (UID: \"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c\") "
Oct 30 19:45:41.876181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875718    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-token\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-token\") pod \"f00f570b-588f-4e4d-839f-423a7fa75cdf\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") "
Oct 30 19:45:41.876181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875756    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-kube-api-access-j4cpf\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-kube-api-access-j4cpf\") pod \"f00f570b-588f-4e4d-839f-423a7fa75cdf\" (UID: \"f00f570b-588f-4e4d-839f-423a7fa75cdf\") "
Oct 30 19:45:41.876181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875784    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-podinfo\") pod \"4212dbf9-58f5-4408-a202-d25f052ded6b\" (UID: \"4212dbf9-58f5-4408-a202-d25f052ded6b\") "
Oct 30 19:45:41.876181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.875798    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-certs" pod "4212dbf9-58f5-4408-a202-d25f052ded6b" (UID: "4212dbf9-58f5-4408-a202-d25f052ded6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.876181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876031    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-data" pod "f00f570b-588f-4e4d-839f-423a7fa75cdf" (UID: "f00f570b-588f-4e4d-839f-423a7fa75cdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.876181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876128    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4212dbf9-58f5-4408-a202-d25f052ded6b-istiod-ca-cert" pod "4212dbf9-58f5-4408-a202-d25f052ded6b" (UID: "4212dbf9-58f5-4408-a202-d25f052ded6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:41.876508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876341    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-socket" pod "f00f570b-588f-4e4d-839f-423a7fa75cdf" (UID: "f00f570b-588f-4e4d-839f-423a7fa75cdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.876571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876525    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-socket" pod "4212dbf9-58f5-4408-a202-d25f052ded6b" (UID: "4212dbf9-58f5-4408-a202-d25f052ded6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.876753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876730    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-data" pod "4212dbf9-58f5-4408-a202-d25f052ded6b" (UID: "4212dbf9-58f5-4408-a202-d25f052ded6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.876940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876917    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876949    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876964    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876978    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.876992    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.877006    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.877020    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212dbf9-58f5-4408-a202-d25f052ded6b-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.877036    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.877050    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877263 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.877062    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.877442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.877270    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-data" pod "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" (UID: "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.879225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.878937    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-envoy" pod "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" (UID: "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.880347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.879987    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-kube-api-access-wdrmx" pod "4212dbf9-58f5-4408-a202-d25f052ded6b" (UID: "4212dbf9-58f5-4408-a202-d25f052ded6b"). InnerVolumeSpecName "kube-api-access-wdrmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:41.880452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.880353    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-envoy" pod "f00f570b-588f-4e4d-839f-423a7fa75cdf" (UID: "f00f570b-588f-4e4d-839f-423a7fa75cdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.881130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.880657    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00f570b-588f-4e4d-839f-423a7fa75cdf-istiod-ca-cert" pod "f00f570b-588f-4e4d-839f-423a7fa75cdf" (UID: "f00f570b-588f-4e4d-839f-423a7fa75cdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:41.881130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.880742    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-podinfo" pod "f00f570b-588f-4e4d-839f-423a7fa75cdf" (UID: "f00f570b-588f-4e4d-839f-423a7fa75cdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:41.881130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.880792    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-podinfo" pod "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" (UID: "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:41.881130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.880874    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istiod-ca-cert" pod "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" (UID: "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:41.881130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.880945    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-podinfo" pod "4212dbf9-58f5-4408-a202-d25f052ded6b" (UID: "4212dbf9-58f5-4408-a202-d25f052ded6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:41.881523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.881492    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-credential-socket" pod "4212dbf9-58f5-4408-a202-d25f052ded6b" (UID: "4212dbf9-58f5-4408-a202-d25f052ded6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.882305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.882280    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-token" pod "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" (UID: "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:41.882391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.882324    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-credential-socket" pod "f00f570b-588f-4e4d-839f-423a7fa75cdf" (UID: "f00f570b-588f-4e4d-839f-423a7fa75cdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.882999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.882968    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-token" pod "4212dbf9-58f5-4408-a202-d25f052ded6b" (UID: "4212dbf9-58f5-4408-a202-d25f052ded6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:41.883076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.883034    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-kube-api-access-j647l" pod "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" (UID: "9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c"). InnerVolumeSpecName "kube-api-access-j647l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:41.883186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.883166    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-envoy" pod "4212dbf9-58f5-4408-a202-d25f052ded6b" (UID: "4212dbf9-58f5-4408-a202-d25f052ded6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:41.885043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.885017    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-kube-api-access-j4cpf" pod "f00f570b-588f-4e4d-839f-423a7fa75cdf" (UID: "f00f570b-588f-4e4d-839f-423a7fa75cdf"). InnerVolumeSpecName "kube-api-access-j4cpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:41.885152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.885060    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-token" pod "f00f570b-588f-4e4d-839f-423a7fa75cdf" (UID: "f00f570b-588f-4e4d-839f-423a7fa75cdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:41.978384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978350    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f00f570b-588f-4e4d-839f-423a7fa75cdf-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978383    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978400    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978412    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdrmx\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-kube-api-access-wdrmx\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978422    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978433    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978441    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978451    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978460    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f00f570b-588f-4e4d-839f-423a7fa75cdf-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978468    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978478    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978486    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4cpf\" (UniqueName: \"kubernetes.io/projected/f00f570b-588f-4e4d-839f-423a7fa75cdf-kube-api-access-j4cpf\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978495    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978503    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212dbf9-58f5-4408-a202-d25f052ded6b-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978512    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978520    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.978571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:41.978528    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j647l\" (UniqueName: \"kubernetes.io/projected/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c-kube-api-access-j647l\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:42.027908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.027866    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"]
Oct 30 19:45:42.032097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.032055    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91366b65-qlrq5"]
Oct 30 19:45:42.095338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.095226    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:42.258283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.258231    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:42.447667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.447625    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"]
Oct 30 19:45:42.448325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448309    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f" containerName="revoke-keys"
Oct 30 19:45:42.448369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448328    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f" containerName="revoke-keys"
Oct 30 19:45:42.448369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448346    4925 cpu_manager.go:414] "containerMap: removing container" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy"
Oct 30 19:45:42.448369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448351    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy"
Oct 30 19:45:42.448369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448361    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy"
Oct 30 19:45:42.448369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448365    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy"
Oct 30 19:45:42.448548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448389    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy"
Oct 30 19:45:42.448548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448395    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy"
Oct 30 19:45:42.448548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448404    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy"
Oct 30 19:45:42.448548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448408    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy"
Oct 30 19:45:42.448548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448533    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f" containerName="revoke-keys"
Oct 30 19:45:42.448548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448543    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerName="istio-proxy"
Oct 30 19:45:42.448766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448550    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy"
Oct 30 19:45:42.448766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448558    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerName="istio-proxy"
Oct 30 19:45:42.448766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.448564    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerName="istio-proxy"
Oct 30 19:45:42.453770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.453752    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.462151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.461489    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c2b092-openshift-default-dockercfg-8hfcl\""
Oct 30 19:45:42.466487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.466464    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"]
Oct 30 19:45:42.472541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.472511    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"]
Oct 30 19:45:42.479068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.479040    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.528649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.526992    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:42.596130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.596091    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-274zz\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-kube-api-access-274zz\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.740896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.740799    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.740896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.740867    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhfdg\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-kube-api-access-bhfdg\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.741152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.740966    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133d072f-aff7-4398-a15f-f065fd99ba14-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.741152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741035    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.741152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741069    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.741152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741095    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.741152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741149    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.741420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741172    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133d072f-aff7-4398-a15f-f065fd99ba14-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.741420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741196    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.741420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741227    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.741420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741342    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.741420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741412    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.741669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741508    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.741669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741582    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.741669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741634    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.742199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741823    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.742199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.741864    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.742199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.714760    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"]
Oct 30 19:45:42.806465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.806433    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:42.809213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.808784    4925 generic.go:356] "Generic (PLEG): container finished" podID="4212dbf9-58f5-4408-a202-d25f052ded6b" containerID="e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5" exitCode=0
Oct 30 19:45:42.809407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.808839    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"
Oct 30 19:45:42.809591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.808853    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" event={"ID":"4212dbf9-58f5-4408-a202-d25f052ded6b","Type":"ContainerDied","Data":"e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5"}
Oct 30 19:45:42.809834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.809800    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs" event={"ID":"4212dbf9-58f5-4408-a202-d25f052ded6b","Type":"ContainerDied","Data":"b1097f668064d1a47af6bdb59bde2208014ebe42eafe7926cbeff27367532657"}
Oct 30 19:45:42.810030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.810005    4925 scope.go:122] "RemoveContainer" containerID="e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5"
Oct 30 19:45:42.811671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.811580    4925 generic.go:356] "Generic (PLEG): container finished" podID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" containerID="634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db" exitCode=0
Oct 30 19:45:42.813839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.813807    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" event={"ID":"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c","Type":"ContainerDied","Data":"634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db"}
Oct 30 19:45:42.813919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.813865    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5" event={"ID":"9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c","Type":"ContainerDied","Data":"96cb0d29c436608ed294382eb75a235b9de392dc841395270e65637fce597079"}
Oct 30 19:45:42.813956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.813946    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"
Oct 30 19:45:42.818313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.818293    4925 generic.go:356] "Generic (PLEG): container finished" podID="f00f570b-588f-4e4d-839f-423a7fa75cdf" containerID="36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c" exitCode=0
Oct 30 19:45:42.818589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.818572    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" event={"ID":"f00f570b-588f-4e4d-839f-423a7fa75cdf","Type":"ContainerDied","Data":"36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c"}
Oct 30 19:45:42.818764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.818750    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"
Oct 30 19:45:42.819622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.818754    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9" event={"ID":"f00f570b-588f-4e4d-839f-423a7fa75cdf","Type":"ContainerDied","Data":"20f3d355f98f3cc1a9db5e16ed2f977a50ab218735cb5f42d514be6c59335c1f"}
Oct 30 19:45:42.828828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.828796    4925 scope.go:122] "RemoveContainer" containerID="e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5"
Oct 30 19:45:42.831888 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:42.831861    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5\": container with ID starting with e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5 not found: ID does not exist" containerID="e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5"
Oct 30 19:45:42.832036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.832013    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5"} err="failed to get container status \"e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5\": rpc error: code = NotFound desc = could not find container \"e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5\": container with ID starting with e0f68aff303ed4b8eb25ae8d4f88b01676fc025d75063e4cc9ed80df22888bc5 not found: ID does not exist"
Oct 30 19:45:42.832132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.832122    4925 scope.go:122] "RemoveContainer" containerID="634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db"
Oct 30 19:45:42.843730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843708    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-274zz\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-kube-api-access-274zz\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.843851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843740    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.843851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843769    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfdg\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-kube-api-access-bhfdg\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.843851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843825    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133d072f-aff7-4398-a15f-f065fd99ba14-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.844011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843865    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.844011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843888    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.844011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843906    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.844011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843937    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.844011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843952    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133d072f-aff7-4398-a15f-f065fd99ba14-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.844011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843968    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.844011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.843986    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.844380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.844050    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.844380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.844080    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.844380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.844131    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.844380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.844203    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.844380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.844232    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.844380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.844319    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.844380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.844337    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.844770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.844635    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.844770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.844757    4925 scope.go:122] "RemoveContainer" containerID="634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db"
Oct 30 19:45:42.845234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.845216    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.845778 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:42.845756    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db\": container with ID starting with 634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db not found: ID does not exist" containerID="634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db"
Oct 30 19:45:42.845955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.845908    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db"} err="failed to get container status \"634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db\": rpc error: code = NotFound desc = could not find container \"634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db\": container with ID starting with 634ebd671c7da3aecfe2f1df022dcc202b80aeb83af4b35c740b715b09d1e2db not found: ID does not exist"
Oct 30 19:45:42.845955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.845955    4925 scope.go:122] "RemoveContainer" containerID="36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c"
Oct 30 19:45:42.846384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.845807    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.846384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.846056    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.846546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.846460    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-certs\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.846783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.846762    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.847486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.847446    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133d072f-aff7-4398-a15f-f065fd99ba14-istiod-ca-cert\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.847762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.847740    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-credential-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.847856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.847821    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"]
Oct 30 19:45:42.848185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.848124    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-data\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.855216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.855196    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-ffzcs"]
Oct 30 19:45:42.857311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.856901    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-socket\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.874783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.864098    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.874783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.872663    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133d072f-aff7-4398-a15f-f065fd99ba14-istio-podinfo\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.877936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.877909    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.878084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.878004    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.878312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.878003    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfdg\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-kube-api-access-bhfdg\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:42.879277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.879262    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-envoy\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.880570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.880545    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-token\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.881384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.881366    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-274zz\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-kube-api-access-274zz\") pod \"e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") " pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:42.883950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.883929    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"]
Oct 30 19:45:42.889455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.889431    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-kmlb5"]
Oct 30 19:45:42.902982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.902960    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"]
Oct 30 19:45:42.917885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.917867    4925 scope.go:122] "RemoveContainer" containerID="374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78"
Oct 30 19:45:42.923737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.923674    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-618d93-openshift-default-79769f779-bhvm9"]
Oct 30 19:45:42.929225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.929207    4925 scope.go:122] "RemoveContainer" containerID="36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c"
Oct 30 19:45:42.929521 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:42.929500    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c\": container with ID starting with 36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c not found: ID does not exist" containerID="36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c"
Oct 30 19:45:42.929608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.929527    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c"} err="failed to get container status \"36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c\": rpc error: code = NotFound desc = could not find container \"36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c\": container with ID starting with 36121e842d48e9ea42ae2de94ed14084a0341b15ebc395dbd2e2b5ea99ea2d7c not found: ID does not exist"
Oct 30 19:45:42.929608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.929545    4925 scope.go:122] "RemoveContainer" containerID="374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78"
Oct 30 19:45:42.934613 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:42.934566    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78\": container with ID starting with 374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78 not found: ID does not exist" containerID="374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78"
Oct 30 19:45:42.934756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:42.934623    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78"} err="failed to get container status \"374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78\": rpc error: code = NotFound desc = could not find container \"374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78\": container with ID starting with 374c3c880c6ad35a5c184b7e46c9e23375a5d3ac54fc99a2518060b6a5d31c78 not found: ID does not exist"
Oct 30 19:45:43.016523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.016457    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:43.072567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.072535    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:43.117876 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:43.117836    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:43.292895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.292765    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:43.300579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.300551    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4212dbf9-58f5-4408-a202-d25f052ded6b" path="/var/lib/kubelet/pods/4212dbf9-58f5-4408-a202-d25f052ded6b/volumes"
Oct 30 19:45:43.301140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.301123    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e391c0-1b15-4f0b-b7c1-6db5d48905c9" path="/var/lib/kubelet/pods/61e391c0-1b15-4f0b-b7c1-6db5d48905c9/volumes"
Oct 30 19:45:43.301597 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.301581    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c" path="/var/lib/kubelet/pods/9117e1cf-fa3d-43a2-969f-7d0baa2a2e6c/volumes"
Oct 30 19:45:43.302733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.302708    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00f570b-588f-4e4d-839f-423a7fa75cdf" path="/var/lib/kubelet/pods/f00f570b-588f-4e4d-839f-423a7fa75cdf/volumes"
Oct 30 19:45:43.320547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.320521    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"]
Oct 30 19:45:43.320807 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:43.320778    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133d072f_aff7_4398_a15f_f065fd99ba14.slice/crio-dd7aa469ed667bc456b236846ffe51456ef2792f99dd636f2006ff16853bcd2e WatchSource:0}: Error finding container dd7aa469ed667bc456b236846ffe51456ef2792f99dd636f2006ff16853bcd2e: Status 404 returned error can't find the container with id dd7aa469ed667bc456b236846ffe51456ef2792f99dd636f2006ff16853bcd2e
Oct 30 19:45:43.324235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.324200    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:43.324317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.324267    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:43.324317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.324296    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:43.376832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.376802    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"]
Oct 30 19:45:43.378191 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:43.378166    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3942c1_19e5_401e_93a1_410c6e6c9f57.slice/crio-4ea5355f77d35e6069ec913399c482649ca6b1100d8a54b41d90b4f2bf1bcfa4 WatchSource:0}: Error finding container 4ea5355f77d35e6069ec913399c482649ca6b1100d8a54b41d90b4f2bf1bcfa4: Status 404 returned error can't find the container with id 4ea5355f77d35e6069ec913399c482649ca6b1100d8a54b41d90b4f2bf1bcfa4
Oct 30 19:45:43.381636 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.381603    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:43.381727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.381666    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:43.381727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.381713    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:43.819372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.819334    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:43.833592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.833560    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" event={"ID":"7a3942c1-19e5-401e-93a1-410c6e6c9f57","Type":"ContainerStarted","Data":"bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d"}
Oct 30 19:45:43.833755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.833610    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" event={"ID":"7a3942c1-19e5-401e-93a1-410c6e6c9f57","Type":"ContainerStarted","Data":"4ea5355f77d35e6069ec913399c482649ca6b1100d8a54b41d90b4f2bf1bcfa4"}
Oct 30 19:45:43.835450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.835421    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" event={"ID":"133d072f-aff7-4398-a15f-f065fd99ba14","Type":"ContainerStarted","Data":"2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5"}
Oct 30 19:45:43.835586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.835468    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" event={"ID":"133d072f-aff7-4398-a15f-f065fd99ba14","Type":"ContainerStarted","Data":"dd7aa469ed667bc456b236846ffe51456ef2792f99dd636f2006ff16853bcd2e"}
Oct 30 19:45:43.859031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.858987    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podStartSLOduration=1.858973231 podStartE2EDuration="1.858973231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:43.855803689 +0000 UTC m=+2378.758816726" watchObservedRunningTime="2026-10-30 19:45:43.858973231 +0000 UTC m=+2378.761986267"
Oct 30 19:45:43.879718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:43.879652    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podStartSLOduration=1.8796397169999999 podStartE2EDuration="1.879639717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:43.877415596 +0000 UTC m=+2378.780428632" watchObservedRunningTime="2026-10-30 19:45:43.879639717 +0000 UTC m=+2378.782652753"
Oct 30 19:45:44.016872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.016843    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:44.017912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.017888    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 19:45:44.018074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.017934    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 19:45:44.072890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.072804    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:44.074103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.074074    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 19:45:44.074280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.074128    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 19:45:44.127965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.124375    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"]
Oct 30 19:45:44.146432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.146406    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.154712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.154545    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-aa49c5-openshift-default-dockercfg-gmnw7\""
Oct 30 19:45:44.206453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.206418    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"]
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.268905    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.297060    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00256e12-693d-485f-b1ba-0f90d0065e73-istiod-ca-cert\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.297196    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-istio-token\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.297291    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-data\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.297344    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00256e12-693d-485f-b1ba-0f90d0065e73-istio-podinfo\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.297402    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-socket\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.297434    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxwh\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-kube-api-access-mmxwh\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.297499    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-envoy\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.297528    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-certs\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.341770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.297666    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-credential-socket\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.531714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.528445    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-istio-token\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.535985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.535954    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-data\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.565141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.565110    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00256e12-693d-485f-b1ba-0f90d0065e73-istio-podinfo\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.574952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.574923    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-socket\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.575931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.575907    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxwh\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-kube-api-access-mmxwh\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.604182 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.604099    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-envoy\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.605621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.605597    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-certs\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.606286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.606268    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-credential-socket\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.606890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.606869    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00256e12-693d-485f-b1ba-0f90d0065e73-istiod-ca-cert\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.786715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.606070    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-certs\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.812823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.564966    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:44.813968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.606610    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-credential-socket\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.828720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.537593    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-data\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.950915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.607611    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00256e12-693d-485f-b1ba-0f90d0065e73-istiod-ca-cert\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.978331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.575479    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-socket\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.978331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.689812    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00256e12-693d-485f-b1ba-0f90d0065e73-istio-podinfo\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.978568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.728968    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxwh\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-kube-api-access-mmxwh\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.978568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.731413    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-istio-token\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.978568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.734676    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-envoy\") pod \"e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") " pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:44.978568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.770895    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"]
Oct 30 19:45:44.986285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.986258    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"]
Oct 30 19:45:44.986428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.986374    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:44.989856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.989835    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:45:44.989997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:44.989875    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:45.009070 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.009035    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:45.017084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.017046    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 19:45:45.017389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.017362    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 19:45:45.072952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.072915    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 19:45:45.073110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.072961    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 19:45:45.094193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.094160    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6br\" (UniqueName: \"kubernetes.io/projected/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-kube-api-access-fr6br\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.094409 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.094306    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.094479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.094443    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.209105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.201920    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6br\" (UniqueName: \"kubernetes.io/projected/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-kube-api-access-fr6br\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.209105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.202035    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.209105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.202128    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.209105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.203266    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.233347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.233281    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.276667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.276572    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6br\" (UniqueName: \"kubernetes.io/projected/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-kube-api-access-fr6br\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.360449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.360172    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:45.444640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.444532    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"]
Oct 30 19:45:45.480049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.479530    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:45.480049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.479606    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:45.480049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.479638    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:45.870859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:45.870821    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:46.040809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:46.040596    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 19:45:46.043507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:46.041037    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 19:45:46.076882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:46.076393    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 19:45:46.076882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:46.076453    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 19:45:46.077283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:46.077236    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"]
Oct 30 19:45:46.168148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:46.167332    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" event={"ID":"00256e12-693d-485f-b1ba-0f90d0065e73","Type":"ContainerStarted","Data":"c02cffb42eed7bae07e2bff7a77513327cc0bc121f8c3d760bdfcc1b9c09347a"}
Oct 30 19:45:46.498480 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:46.487291    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:47.045765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.042774    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 19:45:47.045765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.042847    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 19:45:47.073925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.073786    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 19:45:47.073925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.073862    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 19:45:47.178389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.178350    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" event={"ID":"00256e12-693d-485f-b1ba-0f90d0065e73","Type":"ContainerStarted","Data":"c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f"}
Oct 30 19:45:47.180526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.180337    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf" event={"ID":"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce","Type":"ContainerStarted","Data":"8003a2756d5b3103e336a4460e71cd2b09f0ae29a4824c5d1a08df6aaf3dfd54"}
Oct 30 19:45:47.180526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.180387    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf" event={"ID":"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce","Type":"ContainerStarted","Data":"3d243acd9a167ea5e43b3d36f2215cbec4736a87a32bc8d698d57c4bb4c889be"}
Oct 30 19:45:47.222712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.222195    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podStartSLOduration=3.222174349 podStartE2EDuration="3.222174349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:47.203062584 +0000 UTC m=+2382.106075625" watchObservedRunningTime="2026-10-30 19:45:47.222174349 +0000 UTC m=+2382.125187387"
Oct 30 19:45:47.278551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.278481    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf" podStartSLOduration=3.2784607279999998 podStartE2EDuration="3.278460728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:47.26422767 +0000 UTC m=+2382.167240708" watchObservedRunningTime="2026-10-30 19:45:47.278460728 +0000 UTC m=+2382.181473765"
Oct 30 19:45:47.287466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.287432    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"]
Oct 30 19:45:47.377618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.377536    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"]
Oct 30 19:45:47.463895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.463859    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"]
Oct 30 19:45:47.464071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.464011    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:45:47.469188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.469160    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c2b092\""
Oct 30 19:45:47.469452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.469436    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c2b092-dockercfg-5vjkr\""
Oct 30 19:45:47.578728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.565888    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cn99\" (UniqueName: \"kubernetes.io/projected/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-kube-api-access-9cn99\") pod \"payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd\" (UID: \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:45:47.578728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.566021    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd\" (UID: \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:45:47.671925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.671879    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cn99\" (UniqueName: \"kubernetes.io/projected/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-kube-api-access-9cn99\") pod \"payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd\" (UID: \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:45:47.672668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.672641    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd\" (UID: \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:45:47.677710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.673788    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd\" (UID: \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:45:47.727604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.727572    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cn99\" (UniqueName: \"kubernetes.io/projected/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-kube-api-access-9cn99\") pod \"payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd\" (UID: \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:45:47.803716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:47.801312    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:45:48.031623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.031249    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:48.074755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.072784    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:48.084795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.078495    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 19:45:48.084795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.078547    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 19:45:48.084795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.079307    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 19:45:48.084795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.079348    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 19:45:48.373902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.372917    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:48.386339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.386308    4925 generic.go:356] "Generic (PLEG): container finished" podID="a78fbbda-0cf5-4b8f-9078-68a379dbc3ce" containerID="8003a2756d5b3103e336a4460e71cd2b09f0ae29a4824c5d1a08df6aaf3dfd54" exitCode=0
Oct 30 19:45:48.387217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.386538    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf" event={"ID":"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce","Type":"ContainerDied","Data":"8003a2756d5b3103e336a4460e71cd2b09f0ae29a4824c5d1a08df6aaf3dfd54"}
Oct 30 19:45:48.387454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.387432    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:48.389134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.389086    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:45:48.648776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:48.648742    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"]
Oct 30 19:45:49.056716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.045816    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 19:45:49.056716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.045876    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 19:45:49.074707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.074089    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 19:45:49.074707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.074147    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 19:45:49.348831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.330265    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:49.450375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.450244    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd" event={"ID":"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3","Type":"ContainerStarted","Data":"2324d400d21687ecc0b7aafe4ce1009ce1211bb03d0c1492b546e78ce1a78778"}
Oct 30 19:45:49.450375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.450316    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:45:49.450375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.450342    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd" event={"ID":"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3","Type":"ContainerStarted","Data":"abf66262c2a95239f101330c9b5fa0ebe264e109a516ba1ff34680200c1ef0cd"}
Oct 30 19:45:49.470621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.470570    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd" podStartSLOduration=2.470554619 podStartE2EDuration="2.470554619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:49.468222214 +0000 UTC m=+2384.371235253" watchObservedRunningTime="2026-10-30 19:45:49.470554619 +0000 UTC m=+2384.373567685"
Oct 30 19:45:49.855072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.854456    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:49.879796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.878895    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:49.970440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.969349    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-xks-ca\") pod \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") "
Oct 30 19:45:49.970440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.969404    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-openshift-ca\") pod \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") "
Oct 30 19:45:49.970440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.969475    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-kube-api-access-fr6br\" (UniqueName: \"kubernetes.io/projected/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-kube-api-access-fr6br\") pod \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\" (UID: \"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce\") "
Oct 30 19:45:49.972712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.972657    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-kube-api-access-fr6br" pod "a78fbbda-0cf5-4b8f-9078-68a379dbc3ce" (UID: "a78fbbda-0cf5-4b8f-9078-68a379dbc3ce"). InnerVolumeSpecName "kube-api-access-fr6br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:49.973003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.972982    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-openshift-ca" pod "a78fbbda-0cf5-4b8f-9078-68a379dbc3ce" (UID: "a78fbbda-0cf5-4b8f-9078-68a379dbc3ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:49.975235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:49.975212    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-xks-ca" pod "a78fbbda-0cf5-4b8f-9078-68a379dbc3ce" (UID: "a78fbbda-0cf5-4b8f-9078-68a379dbc3ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:50.051799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.051644    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:50.053506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.053477    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 19:45:50.053717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.053666    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 19:45:50.127776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.089835    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 19:45:50.139813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.139772    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 19:45:50.254739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.247911    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:50.254739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.247953    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:50.254739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.247969    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6br\" (UniqueName: \"kubernetes.io/projected/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce-kube-api-access-fr6br\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:50.329791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.329406    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:50.484713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.484056    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"
Oct 30 19:45:50.484713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.484085    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf" event={"ID":"a78fbbda-0cf5-4b8f-9078-68a379dbc3ce","Type":"ContainerDied","Data":"3d243acd9a167ea5e43b3d36f2215cbec4736a87a32bc8d698d57c4bb4c889be"}
Oct 30 19:45:50.484713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.484144    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d243acd9a167ea5e43b3d36f2215cbec4736a87a32bc8d698d57c4bb4c889be"
Oct 30 19:45:50.933717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:50.933546    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:51.017146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:51.017112    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body=
Oct 30 19:45:51.017337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:51.017174    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused"
Oct 30 19:45:51.073862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:51.073813    4925 patch_prober.go:29] interesting pod/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 30 19:45:51.074064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:51.073885    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 30 19:45:51.173121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:51.172955    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:51.407902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:51.407766    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:52.091842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.091742    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"]
Oct 30 19:45:52.118929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.118848    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b6a735-7hkkp"]
Oct 30 19:45:52.223680 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:52.223619    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:45:52.287907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.287864    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7" podUID="c91cc351-9794-443a-8a92-01b9239e06a4" containerName="payload-pre-processing" containerID="cri-o://a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6" gracePeriod=25
Oct 30 19:45:52.368457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.368314    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:52.369881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.369856    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:52.370434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.370412    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:45:52.441033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.441002    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:52.441372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.441349    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:52.442352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.442331    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:45:52.654895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:52.642677    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:53.324714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:53.324229    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd66596-d69c-4f45-9597-f4d6e6d4f86e" path="/var/lib/kubelet/pods/0cd66596-d69c-4f45-9597-f4d6e6d4f86e/volumes"
Oct 30 19:45:53.397868 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:53.397825    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:53.495510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:53.490642    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:53.508404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:53.508363    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7" podUID="c91cc351-9794-443a-8a92-01b9239e06a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:54.274803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:54.274769    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:54.305592 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:45:54.305550    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:45:54.802839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:54.802792    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:55.001788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:54.983171    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:55.521287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:55.521244    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:55.709287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:55.709250    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:55.823088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:55.823003    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"]
Oct 30 19:45:55.865273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:55.865196    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-e3586f-mp7qm"]
Oct 30 19:45:55.919150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:55.919118    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:56.302020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:56.301986    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:56.878367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:56.877295    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:57.265570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.265537    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f" path="/var/lib/kubelet/pods/d9f94ec1-79c0-4ee5-9a54-5f99971c1b5f/volumes"
Oct 30 19:45:57.328124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.328097    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"]
Oct 30 19:45:57.328898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.328883    4925 cpu_manager.go:414] "containerMap: removing container" podUID="a78fbbda-0cf5-4b8f-9078-68a379dbc3ce" containerName="revoke-keys"
Oct 30 19:45:57.328979 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.328900    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78fbbda-0cf5-4b8f-9078-68a379dbc3ce" containerName="revoke-keys"
Oct 30 19:45:57.329082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.329072    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78fbbda-0cf5-4b8f-9078-68a379dbc3ce" containerName="revoke-keys"
Oct 30 19:45:57.334029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.334011    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.341524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.341497    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:57.341657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.341578    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:45:57.344485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.344458    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"]
Oct 30 19:45:57.401247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.401207    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50648-thj47\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.401435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.401386    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50648-thj47\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.401513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.401469    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jc7\" (UniqueName: \"kubernetes.io/projected/ced34bd7-dda8-4f29-9186-4c65eecac3cf-kube-api-access-w2jc7\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50648-thj47\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.503505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.503472    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jc7\" (UniqueName: \"kubernetes.io/projected/ced34bd7-dda8-4f29-9186-4c65eecac3cf-kube-api-access-w2jc7\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50648-thj47\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.503751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.503728    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50648-thj47\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.503880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.503864    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50648-thj47\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.504481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.504459    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50648-thj47\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.506220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.506201    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50648-thj47\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.515661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.515608    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jc7\" (UniqueName: \"kubernetes.io/projected/ced34bd7-dda8-4f29-9186-4c65eecac3cf-kube-api-access-w2jc7\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50648-thj47\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.659589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.659559    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:45:57.871827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.871739    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:57.944798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:57.944763    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"]
Oct 30 19:45:57.945279 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:57.945249    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced34bd7_dda8_4f29_9186_4c65eecac3cf.slice/crio-15284d493734adda6c6e911210abf4065ff809e3da10bc5d312ed4e189413e62 WatchSource:0}: Error finding container 15284d493734adda6c6e911210abf4065ff809e3da10bc5d312ed4e189413e62: Status 404 returned error can't find the container with id 15284d493734adda6c6e911210abf4065ff809e3da10bc5d312ed4e189413e62
Oct 30 19:45:58.069779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.069736    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"]
Oct 30 19:45:58.077290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.077213    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.086568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.086544    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"]
Oct 30 19:45:58.112972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.112929    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1a844e-openshift-default-dockercfg-465qc\""
Oct 30 19:45:58.114420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.114113    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.114420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.114206    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.114420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.114252    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.114420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.114296    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.114420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.114343    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.114420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.114392    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.114420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.114420    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.114867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.114447    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.114867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.114620    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnrl\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-kube-api-access-wfnrl\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.216355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216326    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.216493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216378    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.216493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216419    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.216665 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216648    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnrl\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-kube-api-access-wfnrl\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.216780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216728    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.216839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216791    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.216839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216806    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.216954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216846    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.216954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216892    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.217062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.216956    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.217118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.217088    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.217251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.217229    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.217313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.217256    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.217448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.217428    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.219432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.219406    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.219548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.219448    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.228299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.228270    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.228425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.228410    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnrl\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-kube-api-access-wfnrl\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.315389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.315363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:58.396164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.396134    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:45:58.641381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.641347    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47" event={"ID":"ced34bd7-dda8-4f29-9186-4c65eecac3cf","Type":"ContainerStarted","Data":"1c72f18374dfdcdce8f816e328f1c18eb2f260529cb88543efbdb1b29335c58d"}
Oct 30 19:45:58.641527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.641409    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47" event={"ID":"ced34bd7-dda8-4f29-9186-4c65eecac3cf","Type":"ContainerStarted","Data":"15284d493734adda6c6e911210abf4065ff809e3da10bc5d312ed4e189413e62"}
Oct 30 19:45:58.658552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.658489    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47" podStartSLOduration=1.658473187 podStartE2EDuration="1.658473187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:58.658186048 +0000 UTC m=+2393.561199086" watchObservedRunningTime="2026-10-30 19:45:58.658473187 +0000 UTC m=+2393.561486225"
Oct 30 19:45:58.692063 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:45:58.692026    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a65bf27_a2e8_4d90_8989_fd9eeae17758.slice/crio-4e4fc67903a3eee4288189bfd74078879112af65da5da4efd878c09a5b5eb506 WatchSource:0}: Error finding container 4e4fc67903a3eee4288189bfd74078879112af65da5da4efd878c09a5b5eb506: Status 404 returned error can't find the container with id 4e4fc67903a3eee4288189bfd74078879112af65da5da4efd878c09a5b5eb506
Oct 30 19:45:58.693036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.693015    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"]
Oct 30 19:45:58.695676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.695637    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:58.695810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.695749    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:58.695810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.695800    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:58.879451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:58.879408    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:59.292949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.292879    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"]
Oct 30 19:45:59.293810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.293773    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy" containerID="cri-o://eeaea7a39894bf375740a962173df0edaf6e23e689a1f9b64df6f93801c4878b" gracePeriod=30
Oct 30 19:45:59.298983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.298959    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"]
Oct 30 19:45:59.299221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.299198    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy" containerID="cri-o://76bebfa0b10ea45244d3d6c58e530524959d040dd6c680d33704d93374ff2f5f" gracePeriod=30
Oct 30 19:45:59.652306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.652267    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" event={"ID":"9a65bf27-a2e8-4d90-8989-fd9eeae17758","Type":"ContainerStarted","Data":"26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32"}
Oct 30 19:45:59.652487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.652325    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" event={"ID":"9a65bf27-a2e8-4d90-8989-fd9eeae17758","Type":"ContainerStarted","Data":"4e4fc67903a3eee4288189bfd74078879112af65da5da4efd878c09a5b5eb506"}
Oct 30 19:45:59.654294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.654264    4925 generic.go:356] "Generic (PLEG): container finished" podID="ced34bd7-dda8-4f29-9186-4c65eecac3cf" containerID="1c72f18374dfdcdce8f816e328f1c18eb2f260529cb88543efbdb1b29335c58d" exitCode=0
Oct 30 19:45:59.654428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.654374    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47" event={"ID":"ced34bd7-dda8-4f29-9186-4c65eecac3cf","Type":"ContainerDied","Data":"1c72f18374dfdcdce8f816e328f1c18eb2f260529cb88543efbdb1b29335c58d"}
Oct 30 19:45:59.674308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.674249    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" podStartSLOduration=1.6742272 podStartE2EDuration="1.6742272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:59.673731248 +0000 UTC m=+2394.576744281" watchObservedRunningTime="2026-10-30 19:45:59.6742272 +0000 UTC m=+2394.577240238"
Oct 30 19:45:59.881016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:45:59.878512    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:00.419899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.419857    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:46:00.422290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.422264    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:46:00.422412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.422324    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:46:00.486822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.486256    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:46:00.566739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.564401    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:00.934814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.934790    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:46:00.963890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.963810    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-xks-ca\") pod \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") "
Oct 30 19:46:00.963890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.963887    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-openshift-ca\") pod \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") "
Oct 30 19:46:00.964113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.964102    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced34bd7-dda8-4f29-9186-4c65eecac3cf-kube-api-access-w2jc7\" (UniqueName: \"kubernetes.io/projected/ced34bd7-dda8-4f29-9186-4c65eecac3cf-kube-api-access-w2jc7\") pod \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\" (UID: \"ced34bd7-dda8-4f29-9186-4c65eecac3cf\") "
Oct 30 19:46:00.964623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.964595    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-openshift-ca" pod "ced34bd7-dda8-4f29-9186-4c65eecac3cf" (UID: "ced34bd7-dda8-4f29-9186-4c65eecac3cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:00.968728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.966930    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-xks-ca" pod "ced34bd7-dda8-4f29-9186-4c65eecac3cf" (UID: "ced34bd7-dda8-4f29-9186-4c65eecac3cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:00.968728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:00.967101    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced34bd7-dda8-4f29-9186-4c65eecac3cf-kube-api-access-w2jc7" pod "ced34bd7-dda8-4f29-9186-4c65eecac3cf" (UID: "ced34bd7-dda8-4f29-9186-4c65eecac3cf"). InnerVolumeSpecName "kube-api-access-w2jc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:01.066758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.066708    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2jc7\" (UniqueName: \"kubernetes.io/projected/ced34bd7-dda8-4f29-9186-4c65eecac3cf-kube-api-access-w2jc7\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:01.066758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.066755    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:01.066758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.066765    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced34bd7-dda8-4f29-9186-4c65eecac3cf-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:01.320912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.320858    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:01.404716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.401063    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:46:01.404716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.401118    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:46:01.491043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.490890    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:01.729113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.729075    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"
Oct 30 19:46:01.729463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.729078    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47" event={"ID":"ced34bd7-dda8-4f29-9186-4c65eecac3cf","Type":"ContainerDied","Data":"15284d493734adda6c6e911210abf4065ff809e3da10bc5d312ed4e189413e62"}
Oct 30 19:46:01.729463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.729438    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15284d493734adda6c6e911210abf4065ff809e3da10bc5d312ed4e189413e62"
Oct 30 19:46:01.940711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:01.939015    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:02.397719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:02.396675    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:46:02.397719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:02.396787    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:46:02.484651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:02.484610    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:02.893769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:02.891201    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:03.399502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:03.399047    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:46:03.399502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:03.399126    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:46:03.504913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:03.503821    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7" podUID="c91cc351-9794-443a-8a92-01b9239e06a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:46:03.513714 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:46:03.509070    4925 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c30400_2461_45d1_a898_8ad75d20331f.slice/crio-74aa127e15085391c4ae9630f749da42ca44d474d28756006815fdc5f68341a7\": RecentStats: unable to find data in memory cache]"
Oct 30 19:46:03.711100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:03.711009    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:04.025772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.024072    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:04.215650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.215617    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:04.352776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.351262    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"]
Oct 30 19:46:04.352776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.351635    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy" containerID="cri-o://6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557" gracePeriod=30
Oct 30 19:46:04.359086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.359052    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"]
Oct 30 19:46:04.359313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.359282    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy" containerID="cri-o://47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb" gracePeriod=30
Oct 30 19:46:04.458713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.458658    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:04.635766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.622374    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:46:04.635766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.622446    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:46:04.659726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.641481    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:04.864142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.859278    4925 generic.go:356] "Generic (PLEG): container finished" podID="dbd88978-804e-48e5-8a76-50588ca147ea" containerID="eeaea7a39894bf375740a962173df0edaf6e23e689a1f9b64df6f93801c4878b" exitCode=0
Oct 30 19:46:04.864142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.859452    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" event={"ID":"dbd88978-804e-48e5-8a76-50588ca147ea","Type":"ContainerDied","Data":"eeaea7a39894bf375740a962173df0edaf6e23e689a1f9b64df6f93801c4878b"}
Oct 30 19:46:04.941949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.941901    4925 generic.go:356] "Generic (PLEG): container finished" podID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerID="76bebfa0b10ea45244d3d6c58e530524959d040dd6c680d33704d93374ff2f5f" exitCode=0
Oct 30 19:46:04.942140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.942035    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" event={"ID":"c60b132e-cabe-4f8e-8e5b-c83f998e9e84","Type":"ContainerDied","Data":"76bebfa0b10ea45244d3d6c58e530524959d040dd6c680d33704d93374ff2f5f"}
Oct 30 19:46:04.942140 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.942086    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:04.991116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.991079    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"]
Oct 30 19:46:04.992277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.992250    4925 cpu_manager.go:414] "containerMap: removing container" podUID="ced34bd7-dda8-4f29-9186-4c65eecac3cf" containerName="revoke-keys"
Oct 30 19:46:04.992277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.992278    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced34bd7-dda8-4f29-9186-4c65eecac3cf" containerName="revoke-keys"
Oct 30 19:46:04.992508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:04.992491    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced34bd7-dda8-4f29-9186-4c65eecac3cf" containerName="revoke-keys"
Oct 30 19:46:05.031252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.029975    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.035303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.035273    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"]
Oct 30 19:46:05.093064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.093036    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:05.093237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.093219    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:46:05.149795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.149763    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlgb\" (UniqueName: \"kubernetes.io/projected/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-kube-api-access-qwlgb\") pod \"maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.150111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.150091    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.150265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.150248    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.151416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.151401    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:46:05.230151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.230094    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:46:05.251609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.251585    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-credential-socket\") pod \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") "
Oct 30 19:46:05.251884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.251865    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-token\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-token\") pod \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") "
Oct 30 19:46:05.252052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.252036    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-envoy\") pod \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") "
Oct 30 19:46:05.253241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.251943    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-credential-socket" pod "c60b132e-cabe-4f8e-8e5b-c83f998e9e84" (UID: "c60b132e-cabe-4f8e-8e5b-c83f998e9e84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.253418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.253212    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-podinfo\") pod \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") "
Oct 30 19:46:05.253736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.253680    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-socket\") pod \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") "
Oct 30 19:46:05.254039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.253993    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-socket" pod "c60b132e-cabe-4f8e-8e5b-c83f998e9e84" (UID: "c60b132e-cabe-4f8e-8e5b-c83f998e9e84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.254231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.254213    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-data\") pod \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") "
Oct 30 19:46:05.256713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.254358    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-data" pod "c60b132e-cabe-4f8e-8e5b-c83f998e9e84" (UID: "c60b132e-cabe-4f8e-8e5b-c83f998e9e84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.256713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.256614    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-kube-api-access-b8dzc\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-kube-api-access-b8dzc\") pod \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") "
Oct 30 19:46:05.256862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.256726    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-certs\") pod \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") "
Oct 30 19:46:05.256932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.256892    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istiod-ca-cert\") pod \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\" (UID: \"c60b132e-cabe-4f8e-8e5b-c83f998e9e84\") "
Oct 30 19:46:05.257800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.257435    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlgb\" (UniqueName: \"kubernetes.io/projected/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-kube-api-access-qwlgb\") pod \"maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.257800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.257594    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.257800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.257612    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-certs" pod "c60b132e-cabe-4f8e-8e5b-c83f998e9e84" (UID: "c60b132e-cabe-4f8e-8e5b-c83f998e9e84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.257800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.257653    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.259150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.258201    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istiod-ca-cert" pod "c60b132e-cabe-4f8e-8e5b-c83f998e9e84" (UID: "c60b132e-cabe-4f8e-8e5b-c83f998e9e84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:05.259150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.258383    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.259150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.258406    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.259150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.258420    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.259150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.258435    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.259150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.258626    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.261519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.261338    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-envoy" pod "c60b132e-cabe-4f8e-8e5b-c83f998e9e84" (UID: "c60b132e-cabe-4f8e-8e5b-c83f998e9e84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.261519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.261385    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-kube-api-access-b8dzc" pod "c60b132e-cabe-4f8e-8e5b-c83f998e9e84" (UID: "c60b132e-cabe-4f8e-8e5b-c83f998e9e84"). InnerVolumeSpecName "kube-api-access-b8dzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:05.261519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.261413    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-token" pod "c60b132e-cabe-4f8e-8e5b-c83f998e9e84" (UID: "c60b132e-cabe-4f8e-8e5b-c83f998e9e84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:05.261519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.261419    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-podinfo" pod "c60b132e-cabe-4f8e-8e5b-c83f998e9e84" (UID: "c60b132e-cabe-4f8e-8e5b-c83f998e9e84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:05.261889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.261671    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.274926 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.274892    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlgb\" (UniqueName: \"kubernetes.io/projected/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-kube-api-access-qwlgb\") pod \"maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.359525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.359497    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-envoy\") pod \"dbd88978-804e-48e5-8a76-50588ca147ea\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") "
Oct 30 19:46:05.359823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.359800    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbd88978-804e-48e5-8a76-50588ca147ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88978-804e-48e5-8a76-50588ca147ea-istio-podinfo\") pod \"dbd88978-804e-48e5-8a76-50588ca147ea\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") "
Oct 30 19:46:05.360084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.360063    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd88978-804e-48e5-8a76-50588ca147ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88978-804e-48e5-8a76-50588ca147ea-istiod-ca-cert\") pod \"dbd88978-804e-48e5-8a76-50588ca147ea\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") "
Oct 30 19:46:05.360250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.360232    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-istio-token\") pod \"dbd88978-804e-48e5-8a76-50588ca147ea\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") "
Oct 30 19:46:05.360388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.360372    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-certs\") pod \"dbd88978-804e-48e5-8a76-50588ca147ea\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") "
Oct 30 19:46:05.360591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.360574    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-kube-api-access-6s77x\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-kube-api-access-6s77x\") pod \"dbd88978-804e-48e5-8a76-50588ca147ea\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") "
Oct 30 19:46:05.360720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.360434    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd88978-804e-48e5-8a76-50588ca147ea-istiod-ca-cert" pod "dbd88978-804e-48e5-8a76-50588ca147ea" (UID: "dbd88978-804e-48e5-8a76-50588ca147ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:05.360866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.360849    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-credential-socket\") pod \"dbd88978-804e-48e5-8a76-50588ca147ea\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") "
Oct 30 19:46:05.361050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.361034    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-socket\") pod \"dbd88978-804e-48e5-8a76-50588ca147ea\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") "
Oct 30 19:46:05.361228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.361211    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-data\") pod \"dbd88978-804e-48e5-8a76-50588ca147ea\" (UID: \"dbd88978-804e-48e5-8a76-50588ca147ea\") "
Oct 30 19:46:05.362999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.362084    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:46:05.362999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.362225    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8dzc\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-kube-api-access-b8dzc\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.362999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.362258    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.362999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.362275    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.362999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.362289    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.362999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.362303    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c60b132e-cabe-4f8e-8e5b-c83f998e9e84-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.362999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.362318    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd88978-804e-48e5-8a76-50588ca147ea-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.362999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.361339    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-certs" pod "dbd88978-804e-48e5-8a76-50588ca147ea" (UID: "dbd88978-804e-48e5-8a76-50588ca147ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.363610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.363535    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-data" pod "dbd88978-804e-48e5-8a76-50588ca147ea" (UID: "dbd88978-804e-48e5-8a76-50588ca147ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.363610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.362824    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-credential-socket" pod "dbd88978-804e-48e5-8a76-50588ca147ea" (UID: "dbd88978-804e-48e5-8a76-50588ca147ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.363610 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.362913    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-envoy" pod "dbd88978-804e-48e5-8a76-50588ca147ea" (UID: "dbd88978-804e-48e5-8a76-50588ca147ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.363853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.363731    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-socket" pod "dbd88978-804e-48e5-8a76-50588ca147ea" (UID: "dbd88978-804e-48e5-8a76-50588ca147ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:05.364410 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.364389    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-kube-api-access-6s77x" pod "dbd88978-804e-48e5-8a76-50588ca147ea" (UID: "dbd88978-804e-48e5-8a76-50588ca147ea"). InnerVolumeSpecName "kube-api-access-6s77x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:05.364571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.364519    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbd88978-804e-48e5-8a76-50588ca147ea-istio-podinfo" pod "dbd88978-804e-48e5-8a76-50588ca147ea" (UID: "dbd88978-804e-48e5-8a76-50588ca147ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:05.365938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.365912    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-istio-token" pod "dbd88978-804e-48e5-8a76-50588ca147ea" (UID: "dbd88978-804e-48e5-8a76-50588ca147ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:05.370748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.370728    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:05.427220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.427181    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 19:46:05.427400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.427232    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 19:46:05.464719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.464645    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s77x\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-kube-api-access-6s77x\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.464719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.464675    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.464719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.464705    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.464719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.464719    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.464719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.464730    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.465118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.464742    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd88978-804e-48e5-8a76-50588ca147ea-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.465118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.464755    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd88978-804e-48e5-8a76-50588ca147ea-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.465118 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.464770    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd88978-804e-48e5-8a76-50588ca147ea-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:05.707384 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:05.707356    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c682eed_77b7_4a71_880d_4ee1f0e5b74e.slice/crio-fe45487547f16e7e74e8c06095db422953f1d90af122fd306758e276e061ad97 WatchSource:0}: Error finding container fe45487547f16e7e74e8c06095db422953f1d90af122fd306758e276e061ad97: Status 404 returned error can't find the container with id fe45487547f16e7e74e8c06095db422953f1d90af122fd306758e276e061ad97
Oct 30 19:46:05.710650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.710617    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"]
Oct 30 19:46:05.907289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.907260    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:05.954366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.954333    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"
Oct 30 19:46:05.954624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.954370    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm" event={"ID":"dbd88978-804e-48e5-8a76-50588ca147ea","Type":"ContainerDied","Data":"5cfac0932b568117d40a61bf8d0743b5480113a076cfaebabaa852c96adc1ede"}
Oct 30 19:46:05.954624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.954446    4925 scope.go:122] "RemoveContainer" containerID="eeaea7a39894bf375740a962173df0edaf6e23e689a1f9b64df6f93801c4878b"
Oct 30 19:46:05.956751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.956681    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7" event={"ID":"c60b132e-cabe-4f8e-8e5b-c83f998e9e84","Type":"ContainerDied","Data":"9369345a519f599776f3df9d010e1f40a3b1f9715b689db76a3249bbc3dc2dac"}
Oct 30 19:46:05.956922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.956855    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"
Oct 30 19:46:05.959540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.959470    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572" event={"ID":"8c682eed-77b7-4a71-880d-4ee1f0e5b74e","Type":"ContainerStarted","Data":"fe45487547f16e7e74e8c06095db422953f1d90af122fd306758e276e061ad97"}
Oct 30 19:46:05.969067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.969050    4925 scope.go:122] "RemoveContainer" containerID="76bebfa0b10ea45244d3d6c58e530524959d040dd6c680d33704d93374ff2f5f"
Oct 30 19:46:05.981286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.981260    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"]
Oct 30 19:46:05.989697 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:05.989662    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-tjls7"]
Oct 30 19:46:06.019601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.019567    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"]
Oct 30 19:46:06.024394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.024325    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-6x6mm"]
Oct 30 19:46:06.389443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.389367    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:46:06.402506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.402479    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:46:06.402985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.402962    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:46:06.403820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.403802    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:46:06.415476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.415458    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:46:06.664362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.664331    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:06.837897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.837870    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:06.971345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.971268    4925 generic.go:356] "Generic (PLEG): container finished" podID="8c682eed-77b7-4a71-880d-4ee1f0e5b74e" containerID="77c69c6414edd471801ab235612bdecf710054b622f57df90d2d143c977b96a0" exitCode=0
Oct 30 19:46:06.971511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:06.971352    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572" event={"ID":"8c682eed-77b7-4a71-880d-4ee1f0e5b74e","Type":"ContainerDied","Data":"77c69c6414edd471801ab235612bdecf710054b622f57df90d2d143c977b96a0"}
Oct 30 19:46:07.244091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.243956    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:07.266934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.266903    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" path="/var/lib/kubelet/pods/c60b132e-cabe-4f8e-8e5b-c83f998e9e84/volumes"
Oct 30 19:46:07.267654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.267621    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" path="/var/lib/kubelet/pods/dbd88978-804e-48e5-8a76-50588ca147ea/volumes"
Oct 30 19:46:07.967473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.967438    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"]
Oct 30 19:46:07.968189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.968173    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy"
Oct 30 19:46:07.968232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.968191    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy"
Oct 30 19:46:07.968298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.968244    4925 cpu_manager.go:414] "containerMap: removing container" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy"
Oct 30 19:46:07.968298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.968252    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy"
Oct 30 19:46:07.968402 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.968392    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60b132e-cabe-4f8e-8e5b-c83f998e9e84" containerName="istio-proxy"
Oct 30 19:46:07.968440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.968413    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd88978-804e-48e5-8a76-50588ca147ea" containerName="istio-proxy"
Oct 30 19:46:07.972459 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.972440    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:07.980066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.980041    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c0a27e\""
Oct 30 19:46:07.981149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:07.981126    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"]
Oct 30 19:46:08.005873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.005840    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:08.011652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.011620    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"]
Oct 30 19:46:08.020627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.020595    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl\" (UID: \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:08.020982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.020957    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-kube-api-access-2c2mm\") pod \"payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl\" (UID: \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:08.024082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.024048    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"]
Oct 30 19:46:08.024202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.024164    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:08.031550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.031520    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c0a27e-dockercfg-n9hqw\""
Oct 30 19:46:08.170757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.169500    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl\" (UID: \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:08.170757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.169678    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-kube-api-access-2c2mm\") pod \"payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl\" (UID: \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:08.210027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.206824    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl\" (UID: \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:08.217386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.217302    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:08.357626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.355465    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-kube-api-access-2c2mm\") pod \"payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl\" (UID: \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:08.384280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.358174    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6365b5-d077-49a0-bbd9-100ecada9d1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl\" (UID: \"db6365b5-d077-49a0-bbd9-100ecada9d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:08.384280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.358801    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcc7v\" (UniqueName: \"kubernetes.io/projected/db6365b5-d077-49a0-bbd9-100ecada9d1b-kube-api-access-pcc7v\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl\" (UID: \"db6365b5-d077-49a0-bbd9-100ecada9d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:08.384280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.359546    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:08.470713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.470615    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:08.494813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.490172    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6365b5-d077-49a0-bbd9-100ecada9d1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl\" (UID: \"db6365b5-d077-49a0-bbd9-100ecada9d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:08.494813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.490612    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcc7v\" (UniqueName: \"kubernetes.io/projected/db6365b5-d077-49a0-bbd9-100ecada9d1b-kube-api-access-pcc7v\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl\" (UID: \"db6365b5-d077-49a0-bbd9-100ecada9d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:08.494813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.492090    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6365b5-d077-49a0-bbd9-100ecada9d1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl\" (UID: \"db6365b5-d077-49a0-bbd9-100ecada9d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:08.513476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.513449    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcc7v\" (UniqueName: \"kubernetes.io/projected/db6365b5-d077-49a0-bbd9-100ecada9d1b-kube-api-access-pcc7v\") pod \"payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl\" (UID: \"db6365b5-d077-49a0-bbd9-100ecada9d1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:08.557940 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.557911    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:08.592648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.592285    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-kube-api-access-qwlgb\" (UniqueName: \"kubernetes.io/projected/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-kube-api-access-qwlgb\") pod \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") "
Oct 30 19:46:08.592648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.592371    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-xks-ca\") pod \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") "
Oct 30 19:46:08.592648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.592437    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-openshift-ca\") pod \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\" (UID: \"8c682eed-77b7-4a71-880d-4ee1f0e5b74e\") "
Oct 30 19:46:08.593451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.593399    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-openshift-ca" pod "8c682eed-77b7-4a71-880d-4ee1f0e5b74e" (UID: "8c682eed-77b7-4a71-880d-4ee1f0e5b74e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:08.595196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.595173    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-xks-ca" pod "8c682eed-77b7-4a71-880d-4ee1f0e5b74e" (UID: "8c682eed-77b7-4a71-880d-4ee1f0e5b74e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:08.595308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.595264    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-kube-api-access-qwlgb" pod "8c682eed-77b7-4a71-880d-4ee1f0e5b74e" (UID: "8c682eed-77b7-4a71-880d-4ee1f0e5b74e"). InnerVolumeSpecName "kube-api-access-qwlgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:08.645194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.645171    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:08.699708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.695411    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwlgb\" (UniqueName: \"kubernetes.io/projected/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-kube-api-access-qwlgb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:08.699708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.695449    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:08.699708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.695467    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c682eed-77b7-4a71-880d-4ee1f0e5b74e-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:08.800006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.799967    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"]
Oct 30 19:46:08.804518 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:08.804493    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f42eb6_90e6_403a_b6b4_04ab67b8fdf8.slice/crio-d7f02b559ef41681173fa99861ad1295ee3d7a0c2c5d3f6a22c7e841d23ba501 WatchSource:0}: Error finding container d7f02b559ef41681173fa99861ad1295ee3d7a0c2c5d3f6a22c7e841d23ba501: Status 404 returned error can't find the container with id d7f02b559ef41681173fa99861ad1295ee3d7a0c2c5d3f6a22c7e841d23ba501
Oct 30 19:46:08.942384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.942356    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"]
Oct 30 19:46:08.943156 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:08.943130    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6365b5_d077_49a0_bbd9_100ecada9d1b.slice/crio-ac3c23b4f6dd326dda9db70466686737c181227d378814bde0ba23514d971b1e WatchSource:0}: Error finding container ac3c23b4f6dd326dda9db70466686737c181227d378814bde0ba23514d971b1e: Status 404 returned error can't find the container with id ac3c23b4f6dd326dda9db70466686737c181227d378814bde0ba23514d971b1e
Oct 30 19:46:08.996419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.996396    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"
Oct 30 19:46:08.996836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.996442    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572" event={"ID":"8c682eed-77b7-4a71-880d-4ee1f0e5b74e","Type":"ContainerDied","Data":"fe45487547f16e7e74e8c06095db422953f1d90af122fd306758e276e061ad97"}
Oct 30 19:46:08.996836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.996499    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe45487547f16e7e74e8c06095db422953f1d90af122fd306758e276e061ad97"
Oct 30 19:46:08.999048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.998767    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl" event={"ID":"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8","Type":"ContainerStarted","Data":"1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0"}
Oct 30 19:46:08.999048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.998830    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl" event={"ID":"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8","Type":"ContainerStarted","Data":"d7f02b559ef41681173fa99861ad1295ee3d7a0c2c5d3f6a22c7e841d23ba501"}
Oct 30 19:46:08.999048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:08.998856    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:09.000222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.000198    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl" event={"ID":"db6365b5-d077-49a0-bbd9-100ecada9d1b","Type":"ContainerStarted","Data":"ac3c23b4f6dd326dda9db70466686737c181227d378814bde0ba23514d971b1e"}
Oct 30 19:46:09.047808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.047758    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl" podStartSLOduration=2.047744321 podStartE2EDuration="2.047744321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:09.044646516 +0000 UTC m=+2403.947659552" watchObservedRunningTime="2026-10-30 19:46:09.047744321 +0000 UTC m=+2403.950757358"
Oct 30 19:46:09.486839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.486645    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:09.675868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.675812    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:46:09.711197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711172    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-token\") pod \"34b7a170-f2b9-4920-8939-5eb2e687e4df\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") "
Oct 30 19:46:09.711368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711288    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-socket\") pod \"34b7a170-f2b9-4920-8939-5eb2e687e4df\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") "
Oct 30 19:46:09.711435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711365    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b7a170-f2b9-4920-8939-5eb2e687e4df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a170-f2b9-4920-8939-5eb2e687e4df-istiod-ca-cert\") pod \"34b7a170-f2b9-4920-8939-5eb2e687e4df\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") "
Oct 30 19:46:09.711435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711420    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-envoy\") pod \"34b7a170-f2b9-4920-8939-5eb2e687e4df\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") "
Oct 30 19:46:09.711645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711612    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-podinfo\") pod \"34b7a170-f2b9-4920-8939-5eb2e687e4df\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") "
Oct 30 19:46:09.711766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711629    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-socket" pod "34b7a170-f2b9-4920-8939-5eb2e687e4df" (UID: "34b7a170-f2b9-4920-8939-5eb2e687e4df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.711766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711707    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-kube-api-access-jg49m\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-kube-api-access-jg49m\") pod \"34b7a170-f2b9-4920-8939-5eb2e687e4df\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") "
Oct 30 19:46:09.711766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711738    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-data\") pod \"34b7a170-f2b9-4920-8939-5eb2e687e4df\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") "
Oct 30 19:46:09.711936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711842    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-certs\") pod \"34b7a170-f2b9-4920-8939-5eb2e687e4df\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") "
Oct 30 19:46:09.711936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711904    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-credential-socket\") pod \"34b7a170-f2b9-4920-8939-5eb2e687e4df\" (UID: \"34b7a170-f2b9-4920-8939-5eb2e687e4df\") "
Oct 30 19:46:09.712390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.711906    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b7a170-f2b9-4920-8939-5eb2e687e4df-istiod-ca-cert" pod "34b7a170-f2b9-4920-8939-5eb2e687e4df" (UID: "34b7a170-f2b9-4920-8939-5eb2e687e4df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:09.712390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.712146    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-data" pod "34b7a170-f2b9-4920-8939-5eb2e687e4df" (UID: "34b7a170-f2b9-4920-8939-5eb2e687e4df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.712390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.712343    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-credential-socket" pod "34b7a170-f2b9-4920-8939-5eb2e687e4df" (UID: "34b7a170-f2b9-4920-8939-5eb2e687e4df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.712390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.712332    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-certs" pod "34b7a170-f2b9-4920-8939-5eb2e687e4df" (UID: "34b7a170-f2b9-4920-8939-5eb2e687e4df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.712892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.712867    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.712989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.712894    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.712989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.712910    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.712989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.712925    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a170-f2b9-4920-8939-5eb2e687e4df-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.712989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.712940    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.713947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.713916    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-token" pod "34b7a170-f2b9-4920-8939-5eb2e687e4df" (UID: "34b7a170-f2b9-4920-8939-5eb2e687e4df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:09.714949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.714930    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-envoy" pod "34b7a170-f2b9-4920-8939-5eb2e687e4df" (UID: "34b7a170-f2b9-4920-8939-5eb2e687e4df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:09.715214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.715185    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-kube-api-access-jg49m" pod "34b7a170-f2b9-4920-8939-5eb2e687e4df" (UID: "34b7a170-f2b9-4920-8939-5eb2e687e4df"). InnerVolumeSpecName "kube-api-access-jg49m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:09.715345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.715322    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-podinfo" pod "34b7a170-f2b9-4920-8939-5eb2e687e4df" (UID: "34b7a170-f2b9-4920-8939-5eb2e687e4df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:09.806914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.806883    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"]
Oct 30 19:46:09.814202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.814171    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.814202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.814203    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg49m\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-kube-api-access-jg49m\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.814423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.814212    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.814423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.814221    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a170-f2b9-4920-8939-5eb2e687e4df-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:09.815510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.815469    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"]
Oct 30 19:46:09.842505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.842473    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"]
Oct 30 19:46:09.975078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.975044    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:46:09.987555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:09.987526    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:10.018532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.018498    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-podinfo\") pod \"8d958656-4816-478b-b8e0-ffcd59d21fc2\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") "
Oct 30 19:46:10.019052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.018656    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-credential-socket\") pod \"8d958656-4816-478b-b8e0-ffcd59d21fc2\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") "
Oct 30 19:46:10.019052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.018772    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-token\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-token\") pod \"8d958656-4816-478b-b8e0-ffcd59d21fc2\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") "
Oct 30 19:46:10.019052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.018893    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-kube-api-access-pwf4f\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-kube-api-access-pwf4f\") pod \"8d958656-4816-478b-b8e0-ffcd59d21fc2\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") "
Oct 30 19:46:10.019052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.018975    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d958656-4816-478b-b8e0-ffcd59d21fc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d958656-4816-478b-b8e0-ffcd59d21fc2-istiod-ca-cert\") pod \"8d958656-4816-478b-b8e0-ffcd59d21fc2\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") "
Oct 30 19:46:10.019052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.019026    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-data\") pod \"8d958656-4816-478b-b8e0-ffcd59d21fc2\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") "
Oct 30 19:46:10.019360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.019127    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-certs\") pod \"8d958656-4816-478b-b8e0-ffcd59d21fc2\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") "
Oct 30 19:46:10.019360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.019221    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-socket\") pod \"8d958656-4816-478b-b8e0-ffcd59d21fc2\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") "
Oct 30 19:46:10.019360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.019291    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-envoy\") pod \"8d958656-4816-478b-b8e0-ffcd59d21fc2\" (UID: \"8d958656-4816-478b-b8e0-ffcd59d21fc2\") "
Oct 30 19:46:10.019535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.019367    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-certs" pod "8d958656-4816-478b-b8e0-ffcd59d21fc2" (UID: "8d958656-4816-478b-b8e0-ffcd59d21fc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:10.020163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.020133    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.020453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.020426    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-credential-socket" pod "8d958656-4816-478b-b8e0-ffcd59d21fc2" (UID: "8d958656-4816-478b-b8e0-ffcd59d21fc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:10.021363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.021319    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d958656-4816-478b-b8e0-ffcd59d21fc2-istiod-ca-cert" pod "8d958656-4816-478b-b8e0-ffcd59d21fc2" (UID: "8d958656-4816-478b-b8e0-ffcd59d21fc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:10.021757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.021679    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-socket" pod "8d958656-4816-478b-b8e0-ffcd59d21fc2" (UID: "8d958656-4816-478b-b8e0-ffcd59d21fc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:10.021936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.021912    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-data" pod "8d958656-4816-478b-b8e0-ffcd59d21fc2" (UID: "8d958656-4816-478b-b8e0-ffcd59d21fc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:10.023220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.023193    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl" event={"ID":"db6365b5-d077-49a0-bbd9-100ecada9d1b","Type":"ContainerStarted","Data":"3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f"}
Oct 30 19:46:10.023312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.023290    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:10.024361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.024329    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-kube-api-access-pwf4f" pod "8d958656-4816-478b-b8e0-ffcd59d21fc2" (UID: "8d958656-4816-478b-b8e0-ffcd59d21fc2"). InnerVolumeSpecName "kube-api-access-pwf4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:10.025607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.025585    4925 generic.go:356] "Generic (PLEG): container finished" podID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerID="6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557" exitCode=0
Oct 30 19:46:10.025882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.025629    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"
Oct 30 19:46:10.026028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.025657    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" event={"ID":"8d958656-4816-478b-b8e0-ffcd59d21fc2","Type":"ContainerDied","Data":"6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557"}
Oct 30 19:46:10.026101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.026059    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4" event={"ID":"8d958656-4816-478b-b8e0-ffcd59d21fc2","Type":"ContainerDied","Data":"f439fecbda4e08ff87890ed44fdc332cf78f74124cd8ee0c1ba597a9e3ac22ed"}
Oct 30 19:46:10.026101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.026092    4925 scope.go:122] "RemoveContainer" containerID="6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557"
Oct 30 19:46:10.026583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.026558    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-envoy" pod "8d958656-4816-478b-b8e0-ffcd59d21fc2" (UID: "8d958656-4816-478b-b8e0-ffcd59d21fc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:10.029173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.029129    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-token" pod "8d958656-4816-478b-b8e0-ffcd59d21fc2" (UID: "8d958656-4816-478b-b8e0-ffcd59d21fc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:10.029365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.029344    4925 generic.go:356] "Generic (PLEG): container finished" podID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerID="47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb" exitCode=0
Oct 30 19:46:10.029454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.029387    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"
Oct 30 19:46:10.029631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.029543    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" event={"ID":"34b7a170-f2b9-4920-8939-5eb2e687e4df","Type":"ContainerDied","Data":"47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb"}
Oct 30 19:46:10.029631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.029602    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6" event={"ID":"34b7a170-f2b9-4920-8939-5eb2e687e4df","Type":"ContainerDied","Data":"72d927553bb9f25fd5399cce8c50d70b68c9a84bef995c4edd41a996a9bfbbc7"}
Oct 30 19:46:10.033426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.033402    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-podinfo" pod "8d958656-4816-478b-b8e0-ffcd59d21fc2" (UID: "8d958656-4816-478b-b8e0-ffcd59d21fc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:10.045734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.045670    4925 scope.go:122] "RemoveContainer" containerID="6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557"
Oct 30 19:46:10.046055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.046007    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl" podStartSLOduration=3.04599412 podStartE2EDuration="3.04599412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:10.040224292 +0000 UTC m=+2404.943237334" watchObservedRunningTime="2026-10-30 19:46:10.04599412 +0000 UTC m=+2404.949007157"
Oct 30 19:46:10.046181 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:46:10.046099    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557\": container with ID starting with 6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557 not found: ID does not exist" containerID="6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557"
Oct 30 19:46:10.046181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.046129    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557"} err="failed to get container status \"6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557\": rpc error: code = NotFound desc = could not find container \"6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557\": container with ID starting with 6ef45dbe099172d92426f4c9e32c6c6ff4b3d1daec6340534f72b5c8957a2557 not found: ID does not exist"
Oct 30 19:46:10.046181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.046153    4925 scope.go:122] "RemoveContainer" containerID="47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb"
Oct 30 19:46:10.062784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.062735    4925 scope.go:122] "RemoveContainer" containerID="47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb"
Oct 30 19:46:10.063271 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:46:10.063079    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb\": container with ID starting with 47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb not found: ID does not exist" containerID="47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb"
Oct 30 19:46:10.063271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.063139    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb"} err="failed to get container status \"47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb\": rpc error: code = NotFound desc = could not find container \"47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb\": container with ID starting with 47a787c72c37f0c93d5423203640a1b843cd3ddd7ff45fe98d5d5486789357fb not found: ID does not exist"
Oct 30 19:46:10.079797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.079764    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"]
Oct 30 19:46:10.084086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.084059    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-lt6t6"]
Oct 30 19:46:10.121471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.121417    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"]
Oct 30 19:46:10.121675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.121586    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.121675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.121612    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.121675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.121629    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.121675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.121646    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.121675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.121659    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.121675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.121672    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwf4f\" (UniqueName: \"kubernetes.io/projected/8d958656-4816-478b-b8e0-ffcd59d21fc2-kube-api-access-pwf4f\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.121954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.121701    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d958656-4816-478b-b8e0-ffcd59d21fc2-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.121954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.121715    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d958656-4816-478b-b8e0-ffcd59d21fc2-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:10.122215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.122199    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8c682eed-77b7-4a71-880d-4ee1f0e5b74e" containerName="revoke-keys"
Oct 30 19:46:10.122267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.122216    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c682eed-77b7-4a71-880d-4ee1f0e5b74e" containerName="revoke-keys"
Oct 30 19:46:10.122267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.122228    4925 cpu_manager.go:414] "containerMap: removing container" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy"
Oct 30 19:46:10.122267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.122233    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy"
Oct 30 19:46:10.124848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.122911    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy"
Oct 30 19:46:10.124848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.122932    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy"
Oct 30 19:46:10.124848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.123441    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c682eed-77b7-4a71-880d-4ee1f0e5b74e" containerName="revoke-keys"
Oct 30 19:46:10.124848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.123456    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" containerName="istio-proxy"
Oct 30 19:46:10.124848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.123494    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" containerName="istio-proxy"
Oct 30 19:46:10.141024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.131942    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:10.141846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.141820    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-826518\""
Oct 30 19:46:10.147143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.147114    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"]
Oct 30 19:46:10.171988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.171956    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:10.223435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.223343    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqvw\" (UniqueName: \"kubernetes.io/projected/30096819-dd6b-480a-b8e4-5f8cd0302ba7-kube-api-access-fbqvw\") pod \"payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn\" (UID: \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:10.223592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.223486    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30096819-dd6b-480a-b8e4-5f8cd0302ba7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn\" (UID: \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:10.325893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.325848    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqvw\" (UniqueName: \"kubernetes.io/projected/30096819-dd6b-480a-b8e4-5f8cd0302ba7-kube-api-access-fbqvw\") pod \"payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn\" (UID: \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:10.367765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.345305    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30096819-dd6b-480a-b8e4-5f8cd0302ba7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn\" (UID: \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:10.368910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.349610    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30096819-dd6b-480a-b8e4-5f8cd0302ba7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn\" (UID: \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:10.458709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.457273    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqvw\" (UniqueName: \"kubernetes.io/projected/30096819-dd6b-480a-b8e4-5f8cd0302ba7-kube-api-access-fbqvw\") pod \"payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn\" (UID: \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:10.471710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.464074    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:10.510518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.510488    4925 scope.go:122] "RemoveContainer" containerID="b18ed36f78f16ae68ddb1a34f6afb084ce4518387c9383efc8b6d2a15ae21020"
Oct 30 19:46:10.549209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.549107    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"]
Oct 30 19:46:10.557919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.557732    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e3586f-openshift-default-976787cfc-48cd4"]
Oct 30 19:46:10.603398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.603378    4925 scope.go:122] "RemoveContainer" containerID="ac8a34a07fe11f8847201996fbdd7ecdac974622b064ef4c15bc6244e0a8d3fc"
Oct 30 19:46:10.873282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.873256    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"]
Oct 30 19:46:10.874180 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:10.874153    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30096819_dd6b_480a_b8e4_5f8cd0302ba7.slice/crio-8309652f08096c48eb2ec849ba5a97f160672d7c1e89fe15d6610321ea6612e1 WatchSource:0}: Error finding container 8309652f08096c48eb2ec849ba5a97f160672d7c1e89fe15d6610321ea6612e1: Status 404 returned error can't find the container with id 8309652f08096c48eb2ec849ba5a97f160672d7c1e89fe15d6610321ea6612e1
Oct 30 19:46:10.938575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:10.938552    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:11.102905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.102168    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn" event={"ID":"30096819-dd6b-480a-b8e4-5f8cd0302ba7","Type":"ContainerStarted","Data":"5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0"}
Oct 30 19:46:11.103547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.103519    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn" event={"ID":"30096819-dd6b-480a-b8e4-5f8cd0302ba7","Type":"ContainerStarted","Data":"8309652f08096c48eb2ec849ba5a97f160672d7c1e89fe15d6610321ea6612e1"}
Oct 30 19:46:11.103795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.103777    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:11.107273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.107230    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:11.120337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.120253    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn" podStartSLOduration=1.120238086 podStartE2EDuration="1.120238086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:11.119532296 +0000 UTC m=+2406.022545347" watchObservedRunningTime="2026-10-30 19:46:11.120238086 +0000 UTC m=+2406.023251114"
Oct 30 19:46:11.266464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.266434    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b7a170-f2b9-4920-8939-5eb2e687e4df" path="/var/lib/kubelet/pods/34b7a170-f2b9-4920-8939-5eb2e687e4df/volumes"
Oct 30 19:46:11.267077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.267056    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d958656-4816-478b-b8e0-ffcd59d21fc2" path="/var/lib/kubelet/pods/8d958656-4816-478b-b8e0-ffcd59d21fc2/volumes"
Oct 30 19:46:11.751524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.751487    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:11.849449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.849411    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"]
Oct 30 19:46:11.854126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:11.854100    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2405fb-qklhf"]
Oct 30 19:46:12.228773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.224214    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:12.405022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.404993    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:12.574825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.574798    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:12.614368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.614338    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:46:12.761845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.761815    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91cc351-9794-443a-8a92-01b9239e06a4-kube-api-access-lfqq8\" (UniqueName: \"kubernetes.io/projected/c91cc351-9794-443a-8a92-01b9239e06a4-kube-api-access-lfqq8\") pod \"c91cc351-9794-443a-8a92-01b9239e06a4\" (UID: \"c91cc351-9794-443a-8a92-01b9239e06a4\") "
Oct 30 19:46:12.762087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.762072    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91cc351-9794-443a-8a92-01b9239e06a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91cc351-9794-443a-8a92-01b9239e06a4-plugins-config-volume\") pod \"c91cc351-9794-443a-8a92-01b9239e06a4\" (UID: \"c91cc351-9794-443a-8a92-01b9239e06a4\") "
Oct 30 19:46:12.762443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.762413    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91cc351-9794-443a-8a92-01b9239e06a4-plugins-config-volume" pod "c91cc351-9794-443a-8a92-01b9239e06a4" (UID: "c91cc351-9794-443a-8a92-01b9239e06a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:12.762736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.762605    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91cc351-9794-443a-8a92-01b9239e06a4-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:12.764308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.764277    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91cc351-9794-443a-8a92-01b9239e06a4-kube-api-access-lfqq8" pod "c91cc351-9794-443a-8a92-01b9239e06a4" (UID: "c91cc351-9794-443a-8a92-01b9239e06a4"). InnerVolumeSpecName "kube-api-access-lfqq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:12.864425 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:12.864345    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfqq8\" (UniqueName: \"kubernetes.io/projected/c91cc351-9794-443a-8a92-01b9239e06a4-kube-api-access-lfqq8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:13.265362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.265330    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78fbbda-0cf5-4b8f-9078-68a379dbc3ce" path="/var/lib/kubelet/pods/a78fbbda-0cf5-4b8f-9078-68a379dbc3ce/volumes"
Oct 30 19:46:13.281164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.281135    4925 generic.go:356] "Generic (PLEG): container finished" podID="c91cc351-9794-443a-8a92-01b9239e06a4" containerID="a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6" exitCode=0
Oct 30 19:46:13.281888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.281865    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"
Oct 30 19:46:13.282472 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.282448    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7" event={"ID":"c91cc351-9794-443a-8a92-01b9239e06a4","Type":"ContainerDied","Data":"a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6"}
Oct 30 19:46:13.282600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.282499    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7" event={"ID":"c91cc351-9794-443a-8a92-01b9239e06a4","Type":"ContainerDied","Data":"25123a449bc5a849d568bf5b953081a5f1087f87d01cef47145bf769e79dfd46"}
Oct 30 19:46:13.282600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.282529    4925 scope.go:122] "RemoveContainer" containerID="a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6"
Oct 30 19:46:13.317710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.313245    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:13.342748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.342718    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"]
Oct 30 19:46:13.349016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.348974    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e3586f-56ff5574c5-vljh7"]
Oct 30 19:46:13.353922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.353899    4925 scope.go:122] "RemoveContainer" containerID="a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6"
Oct 30 19:46:13.354374 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:46:13.354347    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6\": container with ID starting with a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6 not found: ID does not exist" containerID="a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6"
Oct 30 19:46:13.354469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:13.354389    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6"} err="failed to get container status \"a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6\": rpc error: code = NotFound desc = could not find container \"a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6\": container with ID starting with a27826122a28283657c085bd5d24545bacc53691ba5d8aa2217ec995531049d6 not found: ID does not exist"
Oct 30 19:46:14.042057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:14.042023    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:14.683772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:14.683736    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:15.238068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:15.238026    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:15.264018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:15.263988    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91cc351-9794-443a-8a92-01b9239e06a4" path="/var/lib/kubelet/pods/c91cc351-9794-443a-8a92-01b9239e06a4/volumes"
Oct 30 19:46:15.407661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:15.407629    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:15.822661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:15.822569    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:16.459613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:16.459578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:17.059842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:17.059806    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:17.906092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:17.906049    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:18.498711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:18.494538    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:19.180211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:19.179303    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:19.488241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:19.488012    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:19.660193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:19.660138    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:19.750051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:19.749965    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"]
Oct 30 19:46:19.750320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:19.750289    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" containerID="cri-o://e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f" gracePeriod=30
Oct 30 19:46:19.911937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:19.911897    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:20.057236 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.057140    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:20.248614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.248583    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"]
Oct 30 19:46:20.249337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.249320    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c91cc351-9794-443a-8a92-01b9239e06a4" containerName="payload-pre-processing"
Oct 30 19:46:20.249380 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.249339    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91cc351-9794-443a-8a92-01b9239e06a4" containerName="payload-pre-processing"
Oct 30 19:46:20.249471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.249462    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91cc351-9794-443a-8a92-01b9239e06a4" containerName="payload-pre-processing"
Oct 30 19:46:20.254875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.254841    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.258357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.258334    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:46:20.259461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.258855    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:20.263202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.263085    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"]
Oct 30 19:46:20.346083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.338708    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.348473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.348443    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcmz6\" (UniqueName: \"kubernetes.io/projected/730081de-6972-4945-93cf-087f9ae1e9d6-kube-api-access-xcmz6\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.349905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.349879    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.359813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.359708    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:20.451532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.451497    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.451754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.451614    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcmz6\" (UniqueName: \"kubernetes.io/projected/730081de-6972-4945-93cf-087f9ae1e9d6-kube-api-access-xcmz6\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.451824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.451792    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.452376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.452347    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.464726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.461679    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.465159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.465138    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcmz6\" (UniqueName: \"kubernetes.io/projected/730081de-6972-4945-93cf-087f9ae1e9d6-kube-api-access-xcmz6\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.654624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.654590    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:20.940984 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:20.940956    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730081de_6972_4945_93cf_087f9ae1e9d6.slice/crio-94ce2a6cdb56d4c94318f4cb940d222a0346ca162a0eea57ee7c230e539fc9e3 WatchSource:0}: Error finding container 94ce2a6cdb56d4c94318f4cb940d222a0346ca162a0eea57ee7c230e539fc9e3: Status 404 returned error can't find the container with id 94ce2a6cdb56d4c94318f4cb940d222a0346ca162a0eea57ee7c230e539fc9e3
Oct 30 19:46:20.941804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:20.941776    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"]
Oct 30 19:46:21.104162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:21.104138    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:21.376481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:21.376443    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5" event={"ID":"730081de-6972-4945-93cf-087f9ae1e9d6","Type":"ContainerStarted","Data":"5992ee929773159d6129f5a79408fa5869f80b398576e18bfc822997fc9ee90a"}
Oct 30 19:46:21.376666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:21.376512    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5" event={"ID":"730081de-6972-4945-93cf-087f9ae1e9d6","Type":"ContainerStarted","Data":"94ce2a6cdb56d4c94318f4cb940d222a0346ca162a0eea57ee7c230e539fc9e3"}
Oct 30 19:46:21.397226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:21.397172    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5" podStartSLOduration=1.397159379 podStartE2EDuration="1.397159379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:21.392198412 +0000 UTC m=+2416.295211449" watchObservedRunningTime="2026-10-30 19:46:21.397159379 +0000 UTC m=+2416.300172415"
Oct 30 19:46:21.443621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:21.441894    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:22.126338 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:46:22.126298    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:46:22.293077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:22.292992    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:22.389126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:22.389087    4925 generic.go:356] "Generic (PLEG): container finished" podID="730081de-6972-4945-93cf-087f9ae1e9d6" containerID="5992ee929773159d6129f5a79408fa5869f80b398576e18bfc822997fc9ee90a" exitCode=0
Oct 30 19:46:22.389309 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:22.389138    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5" event={"ID":"730081de-6972-4945-93cf-087f9ae1e9d6","Type":"ContainerDied","Data":"5992ee929773159d6129f5a79408fa5869f80b398576e18bfc822997fc9ee90a"}
Oct 30 19:46:22.970725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:22.970606    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:23.332835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.307673    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:46:23.657550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.657523    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:23.730485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.730448    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-openshift-ca\") pod \"730081de-6972-4945-93cf-087f9ae1e9d6\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") "
Oct 30 19:46:23.730752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.730680    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-xks-ca\") pod \"730081de-6972-4945-93cf-087f9ae1e9d6\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") "
Oct 30 19:46:23.730930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.730908    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-openshift-ca" pod "730081de-6972-4945-93cf-087f9ae1e9d6" (UID: "730081de-6972-4945-93cf-087f9ae1e9d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:23.731046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.730936    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730081de-6972-4945-93cf-087f9ae1e9d6-kube-api-access-xcmz6\" (UniqueName: \"kubernetes.io/projected/730081de-6972-4945-93cf-087f9ae1e9d6-kube-api-access-xcmz6\") pod \"730081de-6972-4945-93cf-087f9ae1e9d6\" (UID: \"730081de-6972-4945-93cf-087f9ae1e9d6\") "
Oct 30 19:46:23.731945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.731918    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:23.733676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.733652    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730081de-6972-4945-93cf-087f9ae1e9d6-kube-api-access-xcmz6" pod "730081de-6972-4945-93cf-087f9ae1e9d6" (UID: "730081de-6972-4945-93cf-087f9ae1e9d6"). InnerVolumeSpecName "kube-api-access-xcmz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:23.734595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.734566    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-xks-ca" pod "730081de-6972-4945-93cf-087f9ae1e9d6" (UID: "730081de-6972-4945-93cf-087f9ae1e9d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:23.833643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.833615    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730081de-6972-4945-93cf-087f9ae1e9d6-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:23.833643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.833642    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcmz6\" (UniqueName: \"kubernetes.io/projected/730081de-6972-4945-93cf-087f9ae1e9d6-kube-api-access-xcmz6\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:23.920358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.919919    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:23.966264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.966222    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:23.966473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:23.966289    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:24.425244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:24.425218    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"
Oct 30 19:46:24.425633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:24.425388    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5" event={"ID":"730081de-6972-4945-93cf-087f9ae1e9d6","Type":"ContainerDied","Data":"94ce2a6cdb56d4c94318f4cb940d222a0346ca162a0eea57ee7c230e539fc9e3"}
Oct 30 19:46:24.425633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:24.425429    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94ce2a6cdb56d4c94318f4cb940d222a0346ca162a0eea57ee7c230e539fc9e3"
Oct 30 19:46:24.834769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:24.834723    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main" containerID="cri-o://2c9cd76ed50278c205261994beae1e72b6a582bd7554c8e66b7c7b5703ba9f31" gracePeriod=45
Oct 30 19:46:24.876570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:24.875050    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:24.883582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:24.883514    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" containerID="cri-o://156327d94c00c0a955f24f2678a3f5da15609d7bb91daf53d1c809bf8bf2e6b9" gracePeriod=45
Oct 30 19:46:24.884075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:24.884042    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" containerID="cri-o://aec8bb94d4b544f82ee3a4a0060fc347c14a62f96689b149c472e9ff30837c16" gracePeriod=45
Oct 30 19:46:25.063606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.063375    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:25.370763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.367393    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:46:25.487777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.487742    4925 generic.go:356] "Generic (PLEG): container finished" podID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerID="e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f" exitCode=0
Oct 30 19:46:25.488367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.488119    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"
Oct 30 19:46:25.488994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.488144    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" event={"ID":"99f17c29-b6ba-44b4-beb6-ac220cb960f3","Type":"ContainerDied","Data":"e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f"}
Oct 30 19:46:25.489192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.489173    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp" event={"ID":"99f17c29-b6ba-44b4-beb6-ac220cb960f3","Type":"ContainerDied","Data":"bae077e002c0a2c429b44779e3b5bf6cb7121922f61b325709d442ba1f947a2c"}
Oct 30 19:46:25.489320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.489309    4925 scope.go:122] "RemoveContainer" containerID="e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f"
Oct 30 19:46:25.551045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.551007    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:25.551791 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.551735    4925 generic.go:356] "Generic (PLEG): container finished" podID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerID="aec8bb94d4b544f82ee3a4a0060fc347c14a62f96689b149c472e9ff30837c16" exitCode=0
Oct 30 19:46:25.552034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.551846    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" event={"ID":"527bd7b1-a500-4ea2-ac56-1632216a7d74","Type":"ContainerDied","Data":"aec8bb94d4b544f82ee3a4a0060fc347c14a62f96689b149c472e9ff30837c16"}
Oct 30 19:46:25.552239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.552219    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-kube-api-access-pvldq\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-kube-api-access-pvldq\") pod \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") "
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.552455    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-envoy\") pod \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") "
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.552501    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-token\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-token\") pod \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") "
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.552528    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-podinfo\") pod \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") "
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.552591    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istiod-ca-cert\") pod \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") "
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.552627    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-certs\") pod \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") "
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.552935    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-credential-socket\") pod \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") "
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.553025    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-data\") pod \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") "
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.553058    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-socket\") pod \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\" (UID: \"99f17c29-b6ba-44b4-beb6-ac220cb960f3\") "
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.553670    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-certs" pod "99f17c29-b6ba-44b4-beb6-ac220cb960f3" (UID: "99f17c29-b6ba-44b4-beb6-ac220cb960f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.553864    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.553975    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-socket" pod "99f17c29-b6ba-44b4-beb6-ac220cb960f3" (UID: "99f17c29-b6ba-44b4-beb6-ac220cb960f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.554185    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-data" pod "99f17c29-b6ba-44b4-beb6-ac220cb960f3" (UID: "99f17c29-b6ba-44b4-beb6-ac220cb960f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.554857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.554494    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-credential-socket" pod "99f17c29-b6ba-44b4-beb6-ac220cb960f3" (UID: "99f17c29-b6ba-44b4-beb6-ac220cb960f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.556868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.556455    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istiod-ca-cert" pod "99f17c29-b6ba-44b4-beb6-ac220cb960f3" (UID: "99f17c29-b6ba-44b4-beb6-ac220cb960f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:25.558342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.558307    4925 generic.go:356] "Generic (PLEG): container finished" podID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerID="156327d94c00c0a955f24f2678a3f5da15609d7bb91daf53d1c809bf8bf2e6b9" exitCode=0
Oct 30 19:46:25.558667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.558505    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" event={"ID":"f3cda07a-70d6-468f-8c34-1ce0c64f7653","Type":"ContainerDied","Data":"156327d94c00c0a955f24f2678a3f5da15609d7bb91daf53d1c809bf8bf2e6b9"}
Oct 30 19:46:25.560030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.559994    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-envoy" pod "99f17c29-b6ba-44b4-beb6-ac220cb960f3" (UID: "99f17c29-b6ba-44b4-beb6-ac220cb960f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:25.561103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.561058    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:46:25.562024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.561660    4925 generic.go:356] "Generic (PLEG): container finished" podID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerID="2c9cd76ed50278c205261994beae1e72b6a582bd7554c8e66b7c7b5703ba9f31" exitCode=0
Oct 30 19:46:25.562024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.561865    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" event={"ID":"d434c826-6f85-416c-8270-ccd49d8e38e9","Type":"ContainerDied","Data":"2c9cd76ed50278c205261994beae1e72b6a582bd7554c8e66b7c7b5703ba9f31"}
Oct 30 19:46:25.575647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.575525    4925 scope.go:122] "RemoveContainer" containerID="e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f"
Oct 30 19:46:25.576336 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:46:25.576167    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f\": container with ID starting with e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f not found: ID does not exist" containerID="e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f"
Oct 30 19:46:25.576336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.576206    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f"} err="failed to get container status \"e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f\": rpc error: code = NotFound desc = could not find container \"e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f\": container with ID starting with e92a0783fe410478e7ed22d97fc64f7aa6b29ba46b578fe3bed2385284b85e7f not found: ID does not exist"
Oct 30 19:46:25.576336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.576232    4925 scope.go:122] "RemoveContainer" containerID="2c9cd76ed50278c205261994beae1e72b6a582bd7554c8e66b7c7b5703ba9f31"
Oct 30 19:46:25.578291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.578247    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-token" pod "99f17c29-b6ba-44b4-beb6-ac220cb960f3" (UID: "99f17c29-b6ba-44b4-beb6-ac220cb960f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:25.578574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.578501    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-kube-api-access-pvldq" pod "99f17c29-b6ba-44b4-beb6-ac220cb960f3" (UID: "99f17c29-b6ba-44b4-beb6-ac220cb960f3"). InnerVolumeSpecName "kube-api-access-pvldq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:25.589941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.587986    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-podinfo" pod "99f17c29-b6ba-44b4-beb6-ac220cb960f3" (UID: "99f17c29-b6ba-44b4-beb6-ac220cb960f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:25.589941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.588038    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"]
Oct 30 19:46:25.589941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.588271    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85" podUID="d94a813f-14ef-4507-9628-2af04dd1d1fa" containerName="payload-processing" containerID="cri-o://6a536e06ed201d7842f2adf7745a545000b870b2423be32b9b40d8466405e12f" gracePeriod=30
Oct 30 19:46:25.663115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.656239    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.663115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.656279    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.663115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.656295    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.663115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.656311    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvldq\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-kube-api-access-pvldq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.663115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.656327    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.663115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.656340    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.663115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.656353    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.663115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.656369    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f17c29-b6ba-44b4-beb6-ac220cb960f3-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:25.736036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.735975    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:25.831714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.830926    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d434c826-6f85-416c-8270-ccd49d8e38e9-kube-api-access-4r7cl\" (UniqueName: \"kubernetes.io/projected/d434c826-6f85-416c-8270-ccd49d8e38e9-kube-api-access-4r7cl\") pod \"d434c826-6f85-416c-8270-ccd49d8e38e9\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") "
Oct 30 19:46:25.831714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.831189    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d434c826-6f85-416c-8270-ccd49d8e38e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d434c826-6f85-416c-8270-ccd49d8e38e9-tls-certs\") pod \"d434c826-6f85-416c-8270-ccd49d8e38e9\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") "
Oct 30 19:46:25.831714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.831243    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-home\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-home\") pod \"d434c826-6f85-416c-8270-ccd49d8e38e9\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") "
Oct 30 19:46:25.831714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.831350    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-tmp-dir\") pod \"d434c826-6f85-416c-8270-ccd49d8e38e9\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") "
Oct 30 19:46:25.831714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.831397    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-dshm\") pod \"d434c826-6f85-416c-8270-ccd49d8e38e9\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") "
Oct 30 19:46:25.880726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.869956    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-model-cache\") pod \"d434c826-6f85-416c-8270-ccd49d8e38e9\" (UID: \"d434c826-6f85-416c-8270-ccd49d8e38e9\") "
Oct 30 19:46:25.911958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.879460    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-tmp-dir" pod "d434c826-6f85-416c-8270-ccd49d8e38e9" (UID: "d434c826-6f85-416c-8270-ccd49d8e38e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.065049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.926185    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-home" pod "d434c826-6f85-416c-8270-ccd49d8e38e9" (UID: "d434c826-6f85-416c-8270-ccd49d8e38e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.065049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:25.941232    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:26.162853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.153018    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.162853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.153049    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.163564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.163534    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-model-cache" pod "d434c826-6f85-416c-8270-ccd49d8e38e9" (UID: "d434c826-6f85-416c-8270-ccd49d8e38e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.176835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.176324    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:46:26.225986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.225956    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"]
Oct 30 19:46:26.251404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.251355    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.256376    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-home\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-home\") pod \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") "
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.256418    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tls-certs\") pod \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") "
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.256440    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-dshm\") pod \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") "
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.256611    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-model-cache\") pod \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") "
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.256643    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cda07a-70d6-468f-8c34-1ce0c64f7653-kube-api-access-8866f\" (UniqueName: \"kubernetes.io/projected/f3cda07a-70d6-468f-8c34-1ce0c64f7653-kube-api-access-8866f\") pod \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") "
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.257199    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-home" pod "f3cda07a-70d6-468f-8c34-1ce0c64f7653" (UID: "f3cda07a-70d6-468f-8c34-1ce0c64f7653"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.256682    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tmp-dir\") pod \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\" (UID: \"f3cda07a-70d6-468f-8c34-1ce0c64f7653\") "
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.258476    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.258502    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.258788    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tmp-dir" pod "f3cda07a-70d6-468f-8c34-1ce0c64f7653" (UID: "f3cda07a-70d6-468f-8c34-1ce0c64f7653"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.259002    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-model-cache" pod "f3cda07a-70d6-468f-8c34-1ce0c64f7653" (UID: "f3cda07a-70d6-468f-8c34-1ce0c64f7653"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.260083    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-dshm" pod "f3cda07a-70d6-468f-8c34-1ce0c64f7653" (UID: "f3cda07a-70d6-468f-8c34-1ce0c64f7653"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.262153    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d434c826-6f85-416c-8270-ccd49d8e38e9-kube-api-access-4r7cl" pod "d434c826-6f85-416c-8270-ccd49d8e38e9" (UID: "d434c826-6f85-416c-8270-ccd49d8e38e9"). InnerVolumeSpecName "kube-api-access-4r7cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.265112    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tls-certs" pod "f3cda07a-70d6-468f-8c34-1ce0c64f7653" (UID: "f3cda07a-70d6-468f-8c34-1ce0c64f7653"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.265318    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cda07a-70d6-468f-8c34-1ce0c64f7653-kube-api-access-8866f" pod "f3cda07a-70d6-468f-8c34-1ce0c64f7653" (UID: "f3cda07a-70d6-468f-8c34-1ce0c64f7653"). InnerVolumeSpecName "kube-api-access-8866f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.265624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.265337    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:46:26.275424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.275011    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50648-thj47"]
Oct 30 19:46:26.289264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.289140    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-dshm" pod "d434c826-6f85-416c-8270-ccd49d8e38e9" (UID: "d434c826-6f85-416c-8270-ccd49d8e38e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.289910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.289861    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d434c826-6f85-416c-8270-ccd49d8e38e9-tls-certs" pod "d434c826-6f85-416c-8270-ccd49d8e38e9" (UID: "d434c826-6f85-416c-8270-ccd49d8e38e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:26.335240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.333750    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"]
Oct 30 19:46:26.390726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.372189    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527bd7b1-a500-4ea2-ac56-1632216a7d74-kube-api-access-zhd8l\" (UniqueName: \"kubernetes.io/projected/527bd7b1-a500-4ea2-ac56-1632216a7d74-kube-api-access-zhd8l\") pod \"527bd7b1-a500-4ea2-ac56-1632216a7d74\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") "
Oct 30 19:46:26.390726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.372264    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-tmp-dir\") pod \"527bd7b1-a500-4ea2-ac56-1632216a7d74\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") "
Oct 30 19:46:26.390726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.372397    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-dshm\") pod \"527bd7b1-a500-4ea2-ac56-1632216a7d74\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") "
Oct 30 19:46:26.390726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.372440    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-home\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-home\") pod \"527bd7b1-a500-4ea2-ac56-1632216a7d74\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") "
Oct 30 19:46:26.390726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.372472    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527bd7b1-a500-4ea2-ac56-1632216a7d74-tls-certs\" (UniqueName: \"kubernetes.io/secret/527bd7b1-a500-4ea2-ac56-1632216a7d74-tls-certs\") pod \"527bd7b1-a500-4ea2-ac56-1632216a7d74\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") "
Oct 30 19:46:26.390726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.372567    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-model-cache\") pod \"527bd7b1-a500-4ea2-ac56-1632216a7d74\" (UID: \"527bd7b1-a500-4ea2-ac56-1632216a7d74\") "
Oct 30 19:46:26.456754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.455283    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2405fb-openshift-default-556d785b45-xjxgp"]
Oct 30 19:46:26.457595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.457558    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.457595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.457596    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8866f\" (UniqueName: \"kubernetes.io/projected/f3cda07a-70d6-468f-8c34-1ce0c64f7653-kube-api-access-8866f\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.457829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.457613    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.457829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.457629    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d434c826-6f85-416c-8270-ccd49d8e38e9-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.457829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.457642    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d434c826-6f85-416c-8270-ccd49d8e38e9-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.457829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.457656    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cda07a-70d6-468f-8c34-1ce0c64f7653-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.457829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.457671    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3cda07a-70d6-468f-8c34-1ce0c64f7653-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.457829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.457704    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r7cl\" (UniqueName: \"kubernetes.io/projected/d434c826-6f85-416c-8270-ccd49d8e38e9-kube-api-access-4r7cl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.458639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.458609    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-home" pod "527bd7b1-a500-4ea2-ac56-1632216a7d74" (UID: "527bd7b1-a500-4ea2-ac56-1632216a7d74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.460240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.460212    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-dshm" pod "527bd7b1-a500-4ea2-ac56-1632216a7d74" (UID: "527bd7b1-a500-4ea2-ac56-1632216a7d74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.463718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.460447    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-tmp-dir" pod "527bd7b1-a500-4ea2-ac56-1632216a7d74" (UID: "527bd7b1-a500-4ea2-ac56-1632216a7d74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.464037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.464016    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-model-cache" pod "527bd7b1-a500-4ea2-ac56-1632216a7d74" (UID: "527bd7b1-a500-4ea2-ac56-1632216a7d74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.464757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.464722    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527bd7b1-a500-4ea2-ac56-1632216a7d74-tls-certs" pod "527bd7b1-a500-4ea2-ac56-1632216a7d74" (UID: "527bd7b1-a500-4ea2-ac56-1632216a7d74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:26.482033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.481913    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527bd7b1-a500-4ea2-ac56-1632216a7d74-kube-api-access-zhd8l" pod "527bd7b1-a500-4ea2-ac56-1632216a7d74" (UID: "527bd7b1-a500-4ea2-ac56-1632216a7d74"). InnerVolumeSpecName "kube-api-access-zhd8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.539835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.539573    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:46:26.560224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.560136    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhd8l\" (UniqueName: \"kubernetes.io/projected/527bd7b1-a500-4ea2-ac56-1632216a7d74-kube-api-access-zhd8l\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.560224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.560175    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.560224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.560191    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.560224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.560204    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.560224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.560218    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527bd7b1-a500-4ea2-ac56-1632216a7d74-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.560648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.560242    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/527bd7b1-a500-4ea2-ac56-1632216a7d74-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.574259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.573907    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" event={"ID":"f3cda07a-70d6-468f-8c34-1ce0c64f7653","Type":"ContainerDied","Data":"a0f997d9e9ac4835d92c8e9458832dd09c62c5f08d20f42bec0fb789facc3117"}
Oct 30 19:46:26.574259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.573928    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"
Oct 30 19:46:26.574259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.573961    4925 scope.go:122] "RemoveContainer" containerID="156327d94c00c0a955f24f2678a3f5da15609d7bb91daf53d1c809bf8bf2e6b9"
Oct 30 19:46:26.576887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.576859    4925 generic.go:356] "Generic (PLEG): container finished" podID="d94a813f-14ef-4507-9628-2af04dd1d1fa" containerID="6a536e06ed201d7842f2adf7745a545000b870b2423be32b9b40d8466405e12f" exitCode=0
Oct 30 19:46:26.577020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.576949    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85" event={"ID":"d94a813f-14ef-4507-9628-2af04dd1d1fa","Type":"ContainerDied","Data":"6a536e06ed201d7842f2adf7745a545000b870b2423be32b9b40d8466405e12f"}
Oct 30 19:46:26.578497 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.578479    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"
Oct 30 19:46:26.578773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.578532    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw" event={"ID":"d434c826-6f85-416c-8270-ccd49d8e38e9","Type":"ContainerDied","Data":"b68dbadb146cf592d0d98f6048bc2c3cef3c9e98055a52a340cedebc45af780a"}
Oct 30 19:46:26.583376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.583352    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n" event={"ID":"527bd7b1-a500-4ea2-ac56-1632216a7d74","Type":"ContainerDied","Data":"4857a4c76ac083a6522bafcee17322ac69cd84e6fbfd381f1333a75013bdd0fa"}
Oct 30 19:46:26.583571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.583555    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"
Oct 30 19:46:26.635653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.635611    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:46:26.636017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.635995    4925 scope.go:122] "RemoveContainer" containerID="aec8bb94d4b544f82ee3a4a0060fc347c14a62f96689b149c472e9ff30837c16"
Oct 30 19:46:26.642274 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.642217    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"]
Oct 30 19:46:26.653267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.650131    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/simulator-w1-cfa4bb-kserve-7776c645b-qhhpt"]
Oct 30 19:46:26.655147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.655126    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"]
Oct 30 19:46:26.659609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.659579    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-2-w1-cfa4bb-kserve-6b475f4f79-r7jhw"]
Oct 30 19:46:26.663351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.663328    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"]
Oct 30 19:46:26.668618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.668596    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cfa4bb/distinct-w1-cfa4bb-kserve-846f85d976-4782n"]
Oct 30 19:46:26.685022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.684998    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:46:26.763290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.763256    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94a813f-14ef-4507-9628-2af04dd1d1fa-kube-api-access-9qp9r\" (UniqueName: \"kubernetes.io/projected/d94a813f-14ef-4507-9628-2af04dd1d1fa-kube-api-access-9qp9r\") pod \"d94a813f-14ef-4507-9628-2af04dd1d1fa\" (UID: \"d94a813f-14ef-4507-9628-2af04dd1d1fa\") "
Oct 30 19:46:26.763451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.763348    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d94a813f-14ef-4507-9628-2af04dd1d1fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a813f-14ef-4507-9628-2af04dd1d1fa-plugins-config-volume\") pod \"d94a813f-14ef-4507-9628-2af04dd1d1fa\" (UID: \"d94a813f-14ef-4507-9628-2af04dd1d1fa\") "
Oct 30 19:46:26.763770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.763737    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94a813f-14ef-4507-9628-2af04dd1d1fa-plugins-config-volume" pod "d94a813f-14ef-4507-9628-2af04dd1d1fa" (UID: "d94a813f-14ef-4507-9628-2af04dd1d1fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:26.764121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.764103    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d94a813f-14ef-4507-9628-2af04dd1d1fa-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.765722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.765679    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94a813f-14ef-4507-9628-2af04dd1d1fa-kube-api-access-9qp9r" pod "d94a813f-14ef-4507-9628-2af04dd1d1fa" (UID: "d94a813f-14ef-4507-9628-2af04dd1d1fa"). InnerVolumeSpecName "kube-api-access-9qp9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.821837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.821797    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:26.865939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:26.865852    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qp9r\" (UniqueName: \"kubernetes.io/projected/d94a813f-14ef-4507-9628-2af04dd1d1fa-kube-api-access-9qp9r\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:27.131094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.131002    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:27.272358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.271076    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" path="/var/lib/kubelet/pods/527bd7b1-a500-4ea2-ac56-1632216a7d74/volumes"
Oct 30 19:46:27.272358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.271598    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" path="/var/lib/kubelet/pods/99f17c29-b6ba-44b4-beb6-ac220cb960f3/volumes"
Oct 30 19:46:27.272358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.272057    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced34bd7-dda8-4f29-9186-4c65eecac3cf" path="/var/lib/kubelet/pods/ced34bd7-dda8-4f29-9186-4c65eecac3cf/volumes"
Oct 30 19:46:27.272816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.272798    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" path="/var/lib/kubelet/pods/d434c826-6f85-416c-8270-ccd49d8e38e9/volumes"
Oct 30 19:46:27.273262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.273241    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" path="/var/lib/kubelet/pods/f3cda07a-70d6-468f-8c34-1ce0c64f7653/volumes"
Oct 30 19:46:27.457718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.456819    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:27.479179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.479136    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"]
Oct 30 19:46:27.480091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480063    4925 cpu_manager.go:414] "containerMap: removing container" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main"
Oct 30 19:46:27.480091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480091    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480109    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d94a813f-14ef-4507-9628-2af04dd1d1fa" containerName="payload-processing"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480116    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94a813f-14ef-4507-9628-2af04dd1d1fa" containerName="payload-processing"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480157    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480165    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480176    4925 cpu_manager.go:414] "containerMap: removing container" podUID="730081de-6972-4945-93cf-087f9ae1e9d6" containerName="revoke-keys"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480183    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730081de-6972-4945-93cf-087f9ae1e9d6" containerName="revoke-keys"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480216    4925 cpu_manager.go:414] "containerMap: removing container" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480223    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480256    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main"
Oct 30 19:46:27.480374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480296    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main"
Oct 30 19:46:27.481095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480930    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f17c29-b6ba-44b4-beb6-ac220cb960f3" containerName="istio-proxy"
Oct 30 19:46:27.481095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.480978    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d434c826-6f85-416c-8270-ccd49d8e38e9" containerName="main"
Oct 30 19:46:27.481095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.481005    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="527bd7b1-a500-4ea2-ac56-1632216a7d74" containerName="main"
Oct 30 19:46:27.481095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.481023    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94a813f-14ef-4507-9628-2af04dd1d1fa" containerName="payload-processing"
Oct 30 19:46:27.481095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.481052    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cda07a-70d6-468f-8c34-1ce0c64f7653" containerName="main"
Oct 30 19:46:27.481095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.481067    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="730081de-6972-4945-93cf-087f9ae1e9d6" containerName="revoke-keys"
Oct 30 19:46:27.486326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.486306    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.521743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.501312    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-be6b74-openshift-default-dockercfg-dcrs9\""
Oct 30 19:46:27.541450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.541416    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"]
Oct 30 19:46:27.630608 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.630568    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-token\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.631416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.631389    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx696\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-kube-api-access-nx696\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.631792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.631770    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-podinfo\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.652749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.639437    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:27.689977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.653006    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-socket\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.690389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.688926    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"
Oct 30 19:46:27.691335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.689123    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85" event={"ID":"d94a813f-14ef-4507-9628-2af04dd1d1fa","Type":"ContainerDied","Data":"cc545b2ae2dca93f9f86eacecf14854ed9ac647830c304c0dbb6123be2ffef64"}
Oct 30 19:46:27.691446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.691401    4925 scope.go:122] "RemoveContainer" containerID="6a536e06ed201d7842f2adf7745a545000b870b2423be32b9b40d8466405e12f"
Oct 30 19:46:27.719232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.719148    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istiod-ca-cert\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.719700 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.719634    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-data\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.719885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.719868    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-certs\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.720093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.720060    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-envoy\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.720269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.720254    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-credential-socket\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.823052    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-data\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.823118    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-certs\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.823174    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-envoy\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.823217    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-credential-socket\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.823431    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-token\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.824414    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-data\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.824655    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-credential-socket\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.825233    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-certs\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.828744    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx696\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-kube-api-access-nx696\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.828906    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-podinfo\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.828960    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-socket\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.829056    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istiod-ca-cert\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.835730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.833568    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-socket\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.842572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.842545    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istiod-ca-cert\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.860843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.860815    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"]
Oct 30 19:46:27.910783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.899325    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-token\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.957676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.957633    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50648-549c7ddbb6-67w85"]
Oct 30 19:46:27.963030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.963002    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-podinfo\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.963502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.963480    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-envoy\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:27.974320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:27.974255    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx696\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-kube-api-access-nx696\") pod \"e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") " pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:28.142321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:28.142275    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:28.216713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:28.213121    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:28.439714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:28.438991    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:28.637768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:28.636534    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"]
Oct 30 19:46:28.699558 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:28.699383    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:28.773915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:28.773849    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:28.774198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:28.774172    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:28.774381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:28.774362    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:28.997442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:28.997335    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:29.024381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:29.024297    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" event={"ID":"206b13a6-c1fc-42a0-8f34-01b580c0afbd","Type":"ContainerStarted","Data":"4609f41b91ab3f942144f9dd039ff19317904f58ca02bfcacc3cae1b6ccbd4d3"}
Oct 30 19:46:29.269130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:29.269072    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d94a813f-14ef-4507-9628-2af04dd1d1fa" path="/var/lib/kubelet/pods/d94a813f-14ef-4507-9628-2af04dd1d1fa/volumes"
Oct 30 19:46:30.043203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:30.043156    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" event={"ID":"206b13a6-c1fc-42a0-8f34-01b580c0afbd","Type":"ContainerStarted","Data":"0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d"}
Oct 30 19:46:30.073030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:30.072971    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" podStartSLOduration=3.072956045 podStartE2EDuration="3.072956045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:30.066875948 +0000 UTC m=+2424.969888997" watchObservedRunningTime="2026-10-30 19:46:30.072956045 +0000 UTC m=+2424.975969082"
Oct 30 19:46:30.142957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:30.142927    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:30.144311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:30.144285    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 19:46:30.144432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:30.144342    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 19:46:30.181741 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:30.178663    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:30.527793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:30.527568    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:30.727834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:30.726985    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:31.202716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:31.189954    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:31.376561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:31.376518    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:31.544864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:31.544779    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:32.123560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:32.123525    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:32.124539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:32.124514    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:46:32.561405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:32.561368    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:32.750954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:32.750919    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:33.447492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.447455    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"]
Oct 30 19:46:33.452336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.452308    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"]
Oct 30 19:46:33.452626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.452604    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.456927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.456903    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:33.457171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.457139    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:46:33.457510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.457462    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.462709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.461920    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"]
Oct 30 19:46:33.467704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.467179    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"]
Oct 30 19:46:33.525585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.525541    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14afed5-r5ljk\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.525811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.525606    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a049a861-9x58t\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.525811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.525741    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94n9c\" (UniqueName: \"kubernetes.io/projected/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-kube-api-access-94n9c\") pod \"maas-api-revoke-sub-d14afed5-r5ljk\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.525959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.525860    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a049a861-9x58t\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.525959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.525935    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbpc\" (UniqueName: \"kubernetes.io/projected/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-kube-api-access-qqbpc\") pod \"maas-api-revoke-sub-a049a861-9x58t\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.526245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.526214    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14afed5-r5ljk\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.627573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.627534    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14afed5-r5ljk\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.628035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.627637    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14afed5-r5ljk\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.628035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.627673    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a049a861-9x58t\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.628035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.627726    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n9c\" (UniqueName: \"kubernetes.io/projected/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-kube-api-access-94n9c\") pod \"maas-api-revoke-sub-d14afed5-r5ljk\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.628035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.627788    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a049a861-9x58t\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.628035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.627851    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbpc\" (UniqueName: \"kubernetes.io/projected/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-kube-api-access-qqbpc\") pod \"maas-api-revoke-sub-a049a861-9x58t\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.629068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.628885    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14afed5-r5ljk\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.629068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.628983    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a049a861-9x58t\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.630846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.630823    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a049a861-9x58t\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.635743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.635667    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14afed5-r5ljk\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.639854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.639831    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94n9c\" (UniqueName: \"kubernetes.io/projected/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-kube-api-access-94n9c\") pod \"maas-api-revoke-sub-d14afed5-r5ljk\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.639964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.639880    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbpc\" (UniqueName: \"kubernetes.io/projected/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-kube-api-access-qqbpc\") pod \"maas-api-revoke-sub-a049a861-9x58t\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.792146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.792028    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:33.796633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.796603    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:33.865603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.865422    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:33.877746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.873596    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"]
Oct 30 19:46:33.882198 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.882174    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:33.884878 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.884847    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"]
Oct 30 19:46:33.908816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.891890    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-be6b74\""
Oct 30 19:46:33.908816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.892006    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-be6b74-dockercfg-sdxv8\""
Oct 30 19:46:33.908816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.892466    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"]
Oct 30 19:46:33.908816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.892584    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:33.908816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:33.897998    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"]
Oct 30 19:46:34.092084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.080546    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8f9\" (UniqueName: \"kubernetes.io/projected/c6c6447d-54be-43e8-9402-037e5dc3b1cc-kube-api-access-5p8f9\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft\" (UID: \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:34.092084 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.080967    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6447d-54be-43e8-9402-037e5dc3b1cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft\" (UID: \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:34.238714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.228857    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6447d-54be-43e8-9402-037e5dc3b1cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft\" (UID: \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:34.238714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.228928    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8f9\" (UniqueName: \"kubernetes.io/projected/c6c6447d-54be-43e8-9402-037e5dc3b1cc-kube-api-access-5p8f9\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft\" (UID: \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:34.238714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.231439    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6447d-54be-43e8-9402-037e5dc3b1cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft\" (UID: \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:34.260909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.260840    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8f9\" (UniqueName: \"kubernetes.io/projected/c6c6447d-54be-43e8-9402-037e5dc3b1cc-kube-api-access-5p8f9\") pod \"payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft\" (UID: \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:34.265643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.265598    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:34.330475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.330111    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9679b14b-29ef-48e7-b38b-fcb871083f90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb\" (UID: \"9679b14b-29ef-48e7-b38b-fcb871083f90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:34.330475 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.330231    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhhl\" (UniqueName: \"kubernetes.io/projected/9679b14b-29ef-48e7-b38b-fcb871083f90-kube-api-access-srhhl\") pod \"payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb\" (UID: \"9679b14b-29ef-48e7-b38b-fcb871083f90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:34.431144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.431104    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:46:34.433259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.431174    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:46:34.433259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.432240    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9679b14b-29ef-48e7-b38b-fcb871083f90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb\" (UID: \"9679b14b-29ef-48e7-b38b-fcb871083f90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:34.433259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.432433    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhhl\" (UniqueName: \"kubernetes.io/projected/9679b14b-29ef-48e7-b38b-fcb871083f90-kube-api-access-srhhl\") pod \"payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb\" (UID: \"9679b14b-29ef-48e7-b38b-fcb871083f90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:34.433259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.433032    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9679b14b-29ef-48e7-b38b-fcb871083f90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb\" (UID: \"9679b14b-29ef-48e7-b38b-fcb871083f90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:34.444894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.444867    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhhl\" (UniqueName: \"kubernetes.io/projected/9679b14b-29ef-48e7-b38b-fcb871083f90-kube-api-access-srhhl\") pod \"payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb\" (UID: \"9679b14b-29ef-48e7-b38b-fcb871083f90\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:34.524065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.524002    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:34.549405 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:34.549312    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd628f4ea_91c2_4e94_8c59_76dbf7eccfdf.slice/crio-900f7670fce5f6a665cc21eb8b6e569c76bc4deec49aa2a53deee4d58d0a84d2 WatchSource:0}: Error finding container 900f7670fce5f6a665cc21eb8b6e569c76bc4deec49aa2a53deee4d58d0a84d2: Status 404 returned error can't find the container with id 900f7670fce5f6a665cc21eb8b6e569c76bc4deec49aa2a53deee4d58d0a84d2
Oct 30 19:46:34.554185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.554149    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"]
Oct 30 19:46:34.559247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.559097    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"]
Oct 30 19:46:34.579451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.579416    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:34.705919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.701344    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:34.869284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.866514    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"]
Oct 30 19:46:34.877328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.877147    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:34.969187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:34.969161    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"]
Oct 30 19:46:34.970055 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:34.970026    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9679b14b_29ef_48e7_b38b_fcb871083f90.slice/crio-2f63f24bb2defafc3663b18d7d2f3d456dd614bd3812f971171d4d35cf64bbf9 WatchSource:0}: Error finding container 2f63f24bb2defafc3663b18d7d2f3d456dd614bd3812f971171d4d35cf64bbf9: Status 404 returned error can't find the container with id 2f63f24bb2defafc3663b18d7d2f3d456dd614bd3812f971171d4d35cf64bbf9
Oct 30 19:46:35.213796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.209148    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:35.389957 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.389788    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk" event={"ID":"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf","Type":"ContainerStarted","Data":"900f7670fce5f6a665cc21eb8b6e569c76bc4deec49aa2a53deee4d58d0a84d2"}
Oct 30 19:46:35.401066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.399703    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t" event={"ID":"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a","Type":"ContainerStarted","Data":"109c74776b8b37c795007be56453c9e166dea81b9c53ce1afd7a9da31b6e2062"}
Oct 30 19:46:35.401066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.399779    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t" event={"ID":"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a","Type":"ContainerStarted","Data":"b04abc78fabe366e0487771b71c8169fe1778c4d1c5d50260a76904469a6ed48"}
Oct 30 19:46:35.409748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.408063    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft" event={"ID":"c6c6447d-54be-43e8-9402-037e5dc3b1cc","Type":"ContainerStarted","Data":"bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7"}
Oct 30 19:46:35.409748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.408119    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft" event={"ID":"c6c6447d-54be-43e8-9402-037e5dc3b1cc","Type":"ContainerStarted","Data":"4d5757da0c47b7cb14af695d7c840a94b9b17c7a5ae4fcb3ecedd7da507b77f7"}
Oct 30 19:46:35.409748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.409057    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:35.412144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.411942    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb" event={"ID":"9679b14b-29ef-48e7-b38b-fcb871083f90","Type":"ContainerStarted","Data":"0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087"}
Oct 30 19:46:35.412144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.411990    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb" event={"ID":"9679b14b-29ef-48e7-b38b-fcb871083f90","Type":"ContainerStarted","Data":"2f63f24bb2defafc3663b18d7d2f3d456dd614bd3812f971171d4d35cf64bbf9"}
Oct 30 19:46:35.417215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.416004    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:35.483793 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.483538    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:35.487385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.487329    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t" podStartSLOduration=2.487310183 podStartE2EDuration="2.487310183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:35.422911241 +0000 UTC m=+2430.325924280" watchObservedRunningTime="2026-10-30 19:46:35.487310183 +0000 UTC m=+2430.390323222"
Oct 30 19:46:35.538548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.538487    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft" podStartSLOduration=2.5384727480000002 podStartE2EDuration="2.538472748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:35.533837921 +0000 UTC m=+2430.436850962" watchObservedRunningTime="2026-10-30 19:46:35.538472748 +0000 UTC m=+2430.441485782"
Oct 30 19:46:35.559086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.559026    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb" podStartSLOduration=2.559007517 podStartE2EDuration="2.559007517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:35.557207588 +0000 UTC m=+2430.460220659" watchObservedRunningTime="2026-10-30 19:46:35.559007517 +0000 UTC m=+2430.462020556"
Oct 30 19:46:35.754631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.754544    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:35.936651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:35.936601    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:36.110545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:36.110514    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:36.424172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:36.424139    4925 generic.go:356] "Generic (PLEG): container finished" podID="b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a" containerID="109c74776b8b37c795007be56453c9e166dea81b9c53ce1afd7a9da31b6e2062" exitCode=0
Oct 30 19:46:36.424376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:36.424222    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t" event={"ID":"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a","Type":"ContainerDied","Data":"109c74776b8b37c795007be56453c9e166dea81b9c53ce1afd7a9da31b6e2062"}
Oct 30 19:46:36.426202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:36.426173    4925 generic.go:356] "Generic (PLEG): container finished" podID="d628f4ea-91c2-4e94-8c59-76dbf7eccfdf" containerID="f30f7e2af36929b5a6c6bb0b9b6b5bc85495e953976ab1f70368622d7c3fde3b" exitCode=0
Oct 30 19:46:36.426349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:36.426272    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk" event={"ID":"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf","Type":"ContainerDied","Data":"f30f7e2af36929b5a6c6bb0b9b6b5bc85495e953976ab1f70368622d7c3fde3b"}
Oct 30 19:46:36.669130 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:36.669043    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:37.300007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:37.299381    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:37.526719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:37.524132    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:37.668820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:37.668779    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"]
Oct 30 19:46:37.677128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:37.675659    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-1e7123-78ph5"]
Oct 30 19:46:37.696991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:37.696892    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:37.932057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:37.932024    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:37.967448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:37.967421    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:38.073714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.067098    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-xks-ca\") pod \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") "
Oct 30 19:46:38.073714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.067165    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-openshift-ca\") pod \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") "
Oct 30 19:46:38.073714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.067198    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-kube-api-access-qqbpc\" (UniqueName: \"kubernetes.io/projected/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-kube-api-access-qqbpc\") pod \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\" (UID: \"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a\") "
Oct 30 19:46:38.098943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.086303    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-openshift-ca" pod "b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a" (UID: "b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:38.165077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.164747    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:38.170327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.170294    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:38.175447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.175374    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-xks-ca" pod "b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a" (UID: "b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:38.175826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.175782    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-kube-api-access-qqbpc" pod "b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a" (UID: "b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a"). InnerVolumeSpecName "kube-api-access-qqbpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:38.201919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.201889    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:38.274588 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.274548    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-openshift-ca\") pod \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") "
Oct 30 19:46:38.275188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.274731    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-xks-ca\") pod \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") "
Oct 30 19:46:38.275188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.274868    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-kube-api-access-94n9c\" (UniqueName: \"kubernetes.io/projected/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-kube-api-access-94n9c\") pod \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\" (UID: \"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf\") "
Oct 30 19:46:38.275662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.275492    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-openshift-ca" pod "d628f4ea-91c2-4e94-8c59-76dbf7eccfdf" (UID: "d628f4ea-91c2-4e94-8c59-76dbf7eccfdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:38.276742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.276718    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:38.276920 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.276891    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbpc\" (UniqueName: \"kubernetes.io/projected/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a-kube-api-access-qqbpc\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:38.277194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.276933    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:38.279088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.279061    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-xks-ca" pod "d628f4ea-91c2-4e94-8c59-76dbf7eccfdf" (UID: "d628f4ea-91c2-4e94-8c59-76dbf7eccfdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:38.279189 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.279166    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-kube-api-access-94n9c" pod "d628f4ea-91c2-4e94-8c59-76dbf7eccfdf" (UID: "d628f4ea-91c2-4e94-8c59-76dbf7eccfdf"). InnerVolumeSpecName "kube-api-access-94n9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:38.378523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.378428    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94n9c\" (UniqueName: \"kubernetes.io/projected/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-kube-api-access-94n9c\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:38.379773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.379748    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:38.527883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.527858    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"
Oct 30 19:46:38.528079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.527894    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t" event={"ID":"b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a","Type":"ContainerDied","Data":"b04abc78fabe366e0487771b71c8169fe1778c4d1c5d50260a76904469a6ed48"}
Oct 30 19:46:38.528079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.527949    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04abc78fabe366e0487771b71c8169fe1778c4d1c5d50260a76904469a6ed48"
Oct 30 19:46:38.529812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.529789    4925 generic.go:356] "Generic (PLEG): container finished" podID="00256e12-693d-485f-b1ba-0f90d0065e73" containerID="c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f" exitCode=0
Oct 30 19:46:38.529914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.529863    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" event={"ID":"00256e12-693d-485f-b1ba-0f90d0065e73","Type":"ContainerDied","Data":"c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f"}
Oct 30 19:46:38.530366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.530348    4925 scope.go:122] "RemoveContainer" containerID="c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f"
Oct 30 19:46:38.531486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.531461    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk" event={"ID":"d628f4ea-91c2-4e94-8c59-76dbf7eccfdf","Type":"ContainerDied","Data":"900f7670fce5f6a665cc21eb8b6e569c76bc4deec49aa2a53deee4d58d0a84d2"}
Oct 30 19:46:38.531591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.531510    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="900f7670fce5f6a665cc21eb8b6e569c76bc4deec49aa2a53deee4d58d0a84d2"
Oct 30 19:46:38.531591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.531471    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"
Oct 30 19:46:38.532219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.532192    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:38.532356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.532336    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:38.532430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.532392    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:38.834213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:38.829504    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:39.006965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:39.006931    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:39.171831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:39.171795    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:39.265526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:39.265493    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730081de-6972-4945-93cf-087f9ae1e9d6" path="/var/lib/kubelet/pods/730081de-6972-4945-93cf-087f9ae1e9d6/volumes"
Oct 30 19:46:39.542511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:39.542420    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" event={"ID":"00256e12-693d-485f-b1ba-0f90d0065e73","Type":"ContainerStarted","Data":"a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4"}
Oct 30 19:46:39.668436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:39.668398    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:40.016164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.015754    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:46:40.019884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.019854    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 19:46:40.020087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.019909    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 19:46:40.585221 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.585190    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"]
Oct 30 19:46:40.586363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.586339    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a" containerName="revoke-keys"
Oct 30 19:46:40.586433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.586367    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a" containerName="revoke-keys"
Oct 30 19:46:40.586433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.586411    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d628f4ea-91c2-4e94-8c59-76dbf7eccfdf" containerName="revoke-keys"
Oct 30 19:46:40.586433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.586419    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d628f4ea-91c2-4e94-8c59-76dbf7eccfdf" containerName="revoke-keys"
Oct 30 19:46:40.586623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.586609    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a" containerName="revoke-keys"
Oct 30 19:46:40.586661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.586636    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d628f4ea-91c2-4e94-8c59-76dbf7eccfdf" containerName="revoke-keys"
Oct 30 19:46:40.626753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.591278    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:40.626753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.622597    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:40.626753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.622759    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:46:40.632319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.632288    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"]
Oct 30 19:46:40.742721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.742674    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:40.745324 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.745289    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:40.745491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.745474    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz9kn\" (UniqueName: \"kubernetes.io/projected/51253e1f-197c-4ad8-951f-0a93d314424e-kube-api-access-nz9kn\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:40.745737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.745675    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:40.847193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.847098    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz9kn\" (UniqueName: \"kubernetes.io/projected/51253e1f-197c-4ad8-951f-0a93d314424e-kube-api-access-nz9kn\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:40.847381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.847251    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:40.847381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.847333    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:40.848056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.848037    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:40.849868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.849850    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:40.859276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:40.859246    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz9kn\" (UniqueName: \"kubernetes.io/projected/51253e1f-197c-4ad8-951f-0a93d314424e-kube-api-access-nz9kn\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:41.016091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:41.016054    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 19:46:41.016369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:41.016339    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 19:46:41.054362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:41.054333    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:41.390978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:41.390948    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"]
Oct 30 19:46:41.391734 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:41.391702    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51253e1f_197c_4ad8_951f_0a93d314424e.slice/crio-c6579b01ccb631219cc54a1b4c6984d2db0557eb7fad815674e4e8c19abd979c WatchSource:0}: Error finding container c6579b01ccb631219cc54a1b4c6984d2db0557eb7fad815674e4e8c19abd979c: Status 404 returned error can't find the container with id c6579b01ccb631219cc54a1b4c6984d2db0557eb7fad815674e4e8c19abd979c
Oct 30 19:46:41.561341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:41.561305    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr" event={"ID":"51253e1f-197c-4ad8-951f-0a93d314424e","Type":"ContainerStarted","Data":"c6579b01ccb631219cc54a1b4c6984d2db0557eb7fad815674e4e8c19abd979c"}
Oct 30 19:46:41.618424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:41.616251    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:42.017786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:42.017751    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 19:46:42.018083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:42.018056    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 19:46:42.362783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:42.360513    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:42.571754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:42.571717    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr" event={"ID":"51253e1f-197c-4ad8-951f-0a93d314424e","Type":"ContainerStarted","Data":"8e4d2d13c18f6cf451513a7befd2930196ffcb42ed4922bcc8fb967e2d344383"}
Oct 30 19:46:42.591128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:42.591075    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr" podStartSLOduration=2.591062113 podStartE2EDuration="2.591062113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:42.587805882 +0000 UTC m=+2437.490818920" watchObservedRunningTime="2026-10-30 19:46:42.591062113 +0000 UTC m=+2437.494075151"
Oct 30 19:46:42.971738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:42.968353    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:43.020725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.016095    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 19:46:43.020725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.016159    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 19:46:43.154838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.154808    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"]
Oct 30 19:46:43.160017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.159996    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.174312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.174282    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"]
Oct 30 19:46:43.228803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.228538    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"]
Oct 30 19:46:43.236202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.234026    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.238933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.238910    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"]
Oct 30 19:46:43.289396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.289367    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.289553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.289402    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca278fd-dae1-47a5-8bd4-b350a4820500-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.289553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.289481    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhm9\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-kube-api-access-8zhm9\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.289553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.289507    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.289553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.289544    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.289819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.289658    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.289819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.289762    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.289819 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.289802    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.289972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.289853    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.290080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.290065    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.290122 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.290105    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c707f-dedc-4a62-b3fe-62aa26c6c853-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.290159 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.290124    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.290224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.290208    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.290273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.290251    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.290273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.290269    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.290343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.290309    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b627m\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-kube-api-access-b627m\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.290343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.290335    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.290448 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.290434    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.391978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.391939    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.392179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392019    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.392179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392052    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.392179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392096    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c707f-dedc-4a62-b3fe-62aa26c6c853-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.392360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392221    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.392360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392294    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.392360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392323    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.392360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392350    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.392579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392409    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b627m\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-kube-api-access-b627m\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.392579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392435    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.392579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392490    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.392579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392541    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.392579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392538    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.392579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392570    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.392978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392889    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-certs\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.392978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392891    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-credential-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.392978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392885    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.392978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392958    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca278fd-dae1-47a5-8bd4-b350a4820500-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.392978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.392967    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c707f-dedc-4a62-b3fe-62aa26c6c853-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.393303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.393001    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhm9\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-kube-api-access-8zhm9\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.393303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.393029    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.393303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.393067    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.393303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.393084    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.393303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.393146    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.393303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.393158    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-socket\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.393303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.393232    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.393658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.393420    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-data\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.393967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.393929    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca278fd-dae1-47a5-8bd4-b350a4820500-istiod-ca-cert\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.419607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.395091    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.419607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.419263    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.424573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.424540    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-podinfo\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.424772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.424541    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.426015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.425989    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-envoy\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.426491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.426469    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhm9\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-kube-api-access-8zhm9\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.427617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.427598    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-token\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.428081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.428053    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b627m\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-kube-api-access-b627m\") pod \"e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") " pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.475928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.475905    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:43.552503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.552468    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:43.587268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.587040    4925 generic.go:356] "Generic (PLEG): container finished" podID="51253e1f-197c-4ad8-951f-0a93d314424e" containerID="8e4d2d13c18f6cf451513a7befd2930196ffcb42ed4922bcc8fb967e2d344383" exitCode=0
Oct 30 19:46:43.587268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.587214    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr" event={"ID":"51253e1f-197c-4ad8-951f-0a93d314424e","Type":"ContainerDied","Data":"8e4d2d13c18f6cf451513a7befd2930196ffcb42ed4922bcc8fb967e2d344383"}
Oct 30 19:46:43.830785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.830216    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"]
Oct 30 19:46:43.869721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.867258    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:43.936743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.912195    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:43.936743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.912307    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:43.936743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.912355    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:43.940531 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:43.940486    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca278fd_dae1_47a5_8bd4_b350a4820500.slice/crio-fa9ccdf43f16b2d3c2ce589c8f6bccb87159ed715747cdbfb77ec09fa1be0251 WatchSource:0}: Error finding container fa9ccdf43f16b2d3c2ce589c8f6bccb87159ed715747cdbfb77ec09fa1be0251: Status 404 returned error can't find the container with id fa9ccdf43f16b2d3c2ce589c8f6bccb87159ed715747cdbfb77ec09fa1be0251
Oct 30 19:46:43.942913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.942888    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"]
Oct 30 19:46:43.948848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.948577    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:43.948848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.948638    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:43.948848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:43.948664    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:44.020725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:44.016105    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 19:46:44.020725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:44.016162    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 19:46:44.696732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:44.696353    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" event={"ID":"494c707f-dedc-4a62-b3fe-62aa26c6c853","Type":"ContainerStarted","Data":"e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91"}
Oct 30 19:46:44.696732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:44.696462    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" event={"ID":"494c707f-dedc-4a62-b3fe-62aa26c6c853","Type":"ContainerStarted","Data":"9cf39aea5f4c3a7061cc4f86f55dc49018963bff5a87510fe85da52d2e75d138"}
Oct 30 19:46:44.712715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:44.706145    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" event={"ID":"2ca278fd-dae1-47a5-8bd4-b350a4820500","Type":"ContainerStarted","Data":"f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf"}
Oct 30 19:46:44.712715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:44.706202    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" event={"ID":"2ca278fd-dae1-47a5-8bd4-b350a4820500","Type":"ContainerStarted","Data":"fa9ccdf43f16b2d3c2ce589c8f6bccb87159ed715747cdbfb77ec09fa1be0251"}
Oct 30 19:46:44.869890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:44.865182    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" podStartSLOduration=1.864953491 podStartE2EDuration="1.864953491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:44.752127276 +0000 UTC m=+2439.655140321" watchObservedRunningTime="2026-10-30 19:46:44.864953491 +0000 UTC m=+2439.767966533"
Oct 30 19:46:44.888254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:44.888179    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" podStartSLOduration=1.888161443 podStartE2EDuration="1.888161443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:44.790366298 +0000 UTC m=+2439.693379327" watchObservedRunningTime="2026-10-30 19:46:44.888161443 +0000 UTC m=+2439.791174481"
Oct 30 19:46:45.066759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.066655    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:45.079186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.079156    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Oct 30 19:46:45.080767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.079433    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Oct 30 19:46:45.159537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.159506    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:45.265935 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.265905    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51253e1f-197c-4ad8-951f-0a93d314424e-kube-api-access-nz9kn\" (UniqueName: \"kubernetes.io/projected/51253e1f-197c-4ad8-951f-0a93d314424e-kube-api-access-nz9kn\") pod \"51253e1f-197c-4ad8-951f-0a93d314424e\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") "
Oct 30 19:46:45.266156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.265975    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-xks-ca\") pod \"51253e1f-197c-4ad8-951f-0a93d314424e\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") "
Oct 30 19:46:45.266156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.266058    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-openshift-ca\") pod \"51253e1f-197c-4ad8-951f-0a93d314424e\" (UID: \"51253e1f-197c-4ad8-951f-0a93d314424e\") "
Oct 30 19:46:45.266781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.266737    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-openshift-ca" pod "51253e1f-197c-4ad8-951f-0a93d314424e" (UID: "51253e1f-197c-4ad8-951f-0a93d314424e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:45.268708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.268669    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-xks-ca" pod "51253e1f-197c-4ad8-951f-0a93d314424e" (UID: "51253e1f-197c-4ad8-951f-0a93d314424e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:45.269370 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.269350    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51253e1f-197c-4ad8-951f-0a93d314424e-kube-api-access-nz9kn" pod "51253e1f-197c-4ad8-951f-0a93d314424e" (UID: "51253e1f-197c-4ad8-951f-0a93d314424e"). InnerVolumeSpecName "kube-api-access-nz9kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:45.381042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.380988    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz9kn\" (UniqueName: \"kubernetes.io/projected/51253e1f-197c-4ad8-951f-0a93d314424e-kube-api-access-nz9kn\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:45.381042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.381032    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:45.381042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.381046    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51253e1f-197c-4ad8-951f-0a93d314424e-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:45.476358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.476331    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:45.477615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.477351    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:46:45.477615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.477407    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:46:45.553398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.553319    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:45.555072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.555040    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 19:46:45.555175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.555110    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 19:46:45.745749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.736673    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:45.766713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.760001    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"
Oct 30 19:46:45.766713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.760113    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr" event={"ID":"51253e1f-197c-4ad8-951f-0a93d314424e","Type":"ContainerDied","Data":"c6579b01ccb631219cc54a1b4c6984d2db0557eb7fad815674e4e8c19abd979c"}
Oct 30 19:46:45.766713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.760150    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6579b01ccb631219cc54a1b4c6984d2db0557eb7fad815674e4e8c19abd979c"
Oct 30 19:46:45.960737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:45.960681    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:46.061770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.061036    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:46:46.083728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.074130    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:46:46.088929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.088891    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:46:46.090790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.090747    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"]
Oct 30 19:46:46.094570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.094545    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy" containerID="cri-o://84ac7c5cd4020f1ac6cc7949cda813948e5b9bda3c221a88839026740b721307" gracePeriod=30
Oct 30 19:46:46.175103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.175078    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"]
Oct 30 19:46:46.175564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.175540    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy" containerID="cri-o://bd825c44fb769f6fbe76e51e13938d877492f1bd50ae8e084a8cf3f4f86bdc62" gracePeriod=30
Oct 30 19:46:46.191812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.191396    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:46.507436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.495902    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:46:46.507436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.495964    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:46:46.543806 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.542388    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:46.554609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.554562    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 19:46:46.555007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.554974    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 19:46:46.710720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.707731    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"]
Oct 30 19:46:46.710720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.708812    4925 cpu_manager.go:414] "containerMap: removing container" podUID="51253e1f-197c-4ad8-951f-0a93d314424e" containerName="revoke-keys"
Oct 30 19:46:46.710720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.708832    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51253e1f-197c-4ad8-951f-0a93d314424e" containerName="revoke-keys"
Oct 30 19:46:46.710720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.709090    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="51253e1f-197c-4ad8-951f-0a93d314424e" containerName="revoke-keys"
Oct 30 19:46:46.725718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.714112    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:46.743094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.743066    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"]
Oct 30 19:46:46.749560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.748961    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:46.749560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.749047    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:46.749560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.749088    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d98c9\" (UniqueName: \"kubernetes.io/projected/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-kube-api-access-d98c9\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:46.749870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.749627    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:46.821782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.820360    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:46:46.821782 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.820436    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:46.850802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.850742    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:46.850977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.850887    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:46.850977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.850926    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d98c9\" (UniqueName: \"kubernetes.io/projected/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-kube-api-access-d98c9\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:46.852617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.852583    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:46.854916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.854630    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:46.864136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:46.864109    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d98c9\" (UniqueName: \"kubernetes.io/projected/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-kube-api-access-d98c9\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:47.044019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.043965    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:47.447192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.447111    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"]
Oct 30 19:46:47.480847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.480819    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:46:47.480847 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:47.480846    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ac2c85_ed1e_448e_b75d_99e9b145c8bc.slice/crio-9f7d6cd76ef0095fa6da9aed58950a674f19ba4de77c85074219bdf6714c72d3 WatchSource:0}: Error finding container 9f7d6cd76ef0095fa6da9aed58950a674f19ba4de77c85074219bdf6714c72d3: Status 404 returned error can't find the container with id 9f7d6cd76ef0095fa6da9aed58950a674f19ba4de77c85074219bdf6714c72d3
Oct 30 19:46:47.481057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.480874    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:46:47.486830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.486811    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"]
Oct 30 19:46:47.492622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.492594    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"]
Oct 30 19:46:47.557948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.557800    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 19:46:47.557948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.557875    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 19:46:47.562269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.560647    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:46:47.562269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.561213    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:46:47.567762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.567719    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:47.753517 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.752072    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:47.868713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.865316    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst" event={"ID":"83ac2c85-ed1e-448e-b75d-99e9b145c8bc","Type":"ContainerStarted","Data":"9f7d6cd76ef0095fa6da9aed58950a674f19ba4de77c85074219bdf6714c72d3"}
Oct 30 19:46:47.931288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:47.931258    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:48.098131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:48.098028    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:48.098401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:48.098374    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:48.164867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:48.164830    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:48.476843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:48.476815    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:46:48.477542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:48.477257    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:46:48.552936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:48.552901    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 19:46:48.553135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:48.552951    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 19:46:48.875035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:48.874946    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst" event={"ID":"83ac2c85-ed1e-448e-b75d-99e9b145c8bc","Type":"ContainerStarted","Data":"270aaa28b637541ad2eb3472031a3b6666122143520c6f0430defcaa3f50e94f"}
Oct 30 19:46:48.924715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:48.924250    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst" podStartSLOduration=2.924228494 podStartE2EDuration="2.924228494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:48.893426735 +0000 UTC m=+2443.796439772" watchObservedRunningTime="2026-10-30 19:46:48.924228494 +0000 UTC m=+2443.827241531"
Oct 30 19:46:49.476837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:49.476545    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:46:49.476837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:49.476595    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:46:49.554252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:49.553644    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 19:46:49.554252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:49.553851    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 19:46:49.575461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:49.575424    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:49.885188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:49.885097    4925 generic.go:356] "Generic (PLEG): container finished" podID="83ac2c85-ed1e-448e-b75d-99e9b145c8bc" containerID="270aaa28b637541ad2eb3472031a3b6666122143520c6f0430defcaa3f50e94f" exitCode=0
Oct 30 19:46:49.885188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:49.885156    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst" event={"ID":"83ac2c85-ed1e-448e-b75d-99e9b145c8bc","Type":"ContainerDied","Data":"270aaa28b637541ad2eb3472031a3b6666122143520c6f0430defcaa3f50e94f"}
Oct 30 19:46:50.091701 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:50.091641    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:50.256822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:50.256736    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:50.481143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:50.481109    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:46:50.537854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:50.537739    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:46:50.601727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:50.598385    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 19:46:50.601727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:50.598462    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 19:46:51.342171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.342132    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:51.495059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.493673    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 30 19:46:51.495059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.493750    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 30 19:46:51.514312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.512747    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"]
Oct 30 19:46:51.524865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.522356    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"]
Oct 30 19:46:51.567624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.567434    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 19:46:51.567624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.567487    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 19:46:51.577438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.577386    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:51.703833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.703749    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:51.754774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.754411    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-kube-api-access-d98c9\" (UniqueName: \"kubernetes.io/projected/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-kube-api-access-d98c9\") pod \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") "
Oct 30 19:46:51.754774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.754567    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-xks-ca\") pod \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") "
Oct 30 19:46:51.754774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.754646    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-openshift-ca\") pod \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\" (UID: \"83ac2c85-ed1e-448e-b75d-99e9b145c8bc\") "
Oct 30 19:46:51.757014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.756010    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-openshift-ca" pod "83ac2c85-ed1e-448e-b75d-99e9b145c8bc" (UID: "83ac2c85-ed1e-448e-b75d-99e9b145c8bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:51.760821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.757196    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:51.782720 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.779263    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-xks-ca" pod "83ac2c85-ed1e-448e-b75d-99e9b145c8bc" (UID: "83ac2c85-ed1e-448e-b75d-99e9b145c8bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:51.798516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.796032    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-kube-api-access-d98c9" pod "83ac2c85-ed1e-448e-b75d-99e9b145c8bc" (UID: "83ac2c85-ed1e-448e-b75d-99e9b145c8bc"). InnerVolumeSpecName "kube-api-access-d98c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:51.865717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.862208    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d98c9\" (UniqueName: \"kubernetes.io/projected/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-kube-api-access-d98c9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:51.865717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.862257    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:51.865717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:51.862275    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ac2c85-ed1e-448e-b75d-99e9b145c8bc-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.027450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.027419    4925 generic.go:356] "Generic (PLEG): container finished" podID="68b50881-1ed8-48b6-9b45-44146b46a135" containerID="84ac7c5cd4020f1ac6cc7949cda813948e5b9bda3c221a88839026740b721307" exitCode=0
Oct 30 19:46:52.027639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.027486    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" event={"ID":"68b50881-1ed8-48b6-9b45-44146b46a135","Type":"ContainerDied","Data":"84ac7c5cd4020f1ac6cc7949cda813948e5b9bda3c221a88839026740b721307"}
Oct 30 19:46:52.027639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.027545    4925 scope.go:122] "RemoveContainer" containerID="cff9888bdeb8e4abb55c5f768029b401d41054d39d3749a72a5a7e05d6414137"
Oct 30 19:46:52.030141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.030115    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:46:52.031138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.030945    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"
Oct 30 19:46:52.031138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.030955    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst" event={"ID":"83ac2c85-ed1e-448e-b75d-99e9b145c8bc","Type":"ContainerDied","Data":"9f7d6cd76ef0095fa6da9aed58950a674f19ba4de77c85074219bdf6714c72d3"}
Oct 30 19:46:52.031138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.030998    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f7d6cd76ef0095fa6da9aed58950a674f19ba4de77c85074219bdf6714c72d3"
Oct 30 19:46:52.040428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.040396    4925 generic.go:356] "Generic (PLEG): container finished" podID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerID="bd825c44fb769f6fbe76e51e13938d877492f1bd50ae8e084a8cf3f4f86bdc62" exitCode=0
Oct 30 19:46:52.040845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.040744    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497" event={"ID":"cba538c7-458e-42de-84b0-54d9e11fdcd7","Type":"ContainerDied","Data":"bd825c44fb769f6fbe76e51e13938d877492f1bd50ae8e084a8cf3f4f86bdc62"}
Oct 30 19:46:52.041325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.041305    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"
Oct 30 19:46:52.052275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.049036    4925 scope.go:122] "RemoveContainer" containerID="bd825c44fb769f6fbe76e51e13938d877492f1bd50ae8e084a8cf3f4f86bdc62"
Oct 30 19:46:52.101630 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.101533    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.119637    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-kube-api-access-5dc86\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-kube-api-access-5dc86\") pod \"cba538c7-458e-42de-84b0-54d9e11fdcd7\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") "
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.119737    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba538c7-458e-42de-84b0-54d9e11fdcd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba538c7-458e-42de-84b0-54d9e11fdcd7-istiod-ca-cert\") pod \"cba538c7-458e-42de-84b0-54d9e11fdcd7\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") "
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.119783    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-podinfo\") pod \"cba538c7-458e-42de-84b0-54d9e11fdcd7\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") "
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.119915    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-socket\") pod \"cba538c7-458e-42de-84b0-54d9e11fdcd7\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") "
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.124403    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba538c7-458e-42de-84b0-54d9e11fdcd7-istiod-ca-cert" pod "cba538c7-458e-42de-84b0-54d9e11fdcd7" (UID: "cba538c7-458e-42de-84b0-54d9e11fdcd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.125133    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-credential-socket\") pod \"cba538c7-458e-42de-84b0-54d9e11fdcd7\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") "
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.125167    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-certs\") pod \"cba538c7-458e-42de-84b0-54d9e11fdcd7\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") "
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.125199    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-data\") pod \"cba538c7-458e-42de-84b0-54d9e11fdcd7\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") "
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.125293    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-envoy\") pod \"cba538c7-458e-42de-84b0-54d9e11fdcd7\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") "
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.125340    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-token\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-token\") pod \"cba538c7-458e-42de-84b0-54d9e11fdcd7\" (UID: \"cba538c7-458e-42de-84b0-54d9e11fdcd7\") "
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.126004    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba538c7-458e-42de-84b0-54d9e11fdcd7-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.126544    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-credential-socket" pod "cba538c7-458e-42de-84b0-54d9e11fdcd7" (UID: "cba538c7-458e-42de-84b0-54d9e11fdcd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.127549    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-socket" pod "cba538c7-458e-42de-84b0-54d9e11fdcd7" (UID: "cba538c7-458e-42de-84b0-54d9e11fdcd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.127644    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-data" pod "cba538c7-458e-42de-84b0-54d9e11fdcd7" (UID: "cba538c7-458e-42de-84b0-54d9e11fdcd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.128149    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-certs" pod "cba538c7-458e-42de-84b0-54d9e11fdcd7" (UID: "cba538c7-458e-42de-84b0-54d9e11fdcd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.130716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.130476    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-envoy" pod "cba538c7-458e-42de-84b0-54d9e11fdcd7" (UID: "cba538c7-458e-42de-84b0-54d9e11fdcd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.145251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.145215    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-kube-api-access-5dc86" pod "cba538c7-458e-42de-84b0-54d9e11fdcd7" (UID: "cba538c7-458e-42de-84b0-54d9e11fdcd7"). InnerVolumeSpecName "kube-api-access-5dc86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.150254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.150215    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-token" pod "cba538c7-458e-42de-84b0-54d9e11fdcd7" (UID: "cba538c7-458e-42de-84b0-54d9e11fdcd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.150825 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.150765    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-podinfo" pod "cba538c7-458e-42de-84b0-54d9e11fdcd7" (UID: "cba538c7-458e-42de-84b0-54d9e11fdcd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:52.167897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.164108    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:46:52.180141 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:46:52.178956    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:46:52.228747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.228705    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.228747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.228747    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dc86\" (UniqueName: \"kubernetes.io/projected/cba538c7-458e-42de-84b0-54d9e11fdcd7-kube-api-access-5dc86\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.228747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.228766    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.228747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.228780    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.228747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.228794    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.229792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.228808    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.229792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.228822    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.229792 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.228835    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba538c7-458e-42de-84b0-54d9e11fdcd7-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.329645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.329559    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-certs\") pod \"68b50881-1ed8-48b6-9b45-44146b46a135\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") "
Oct 30 19:46:52.329864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.329680    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-envoy\") pod \"68b50881-1ed8-48b6-9b45-44146b46a135\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") "
Oct 30 19:46:52.329864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.329764    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b50881-1ed8-48b6-9b45-44146b46a135-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b50881-1ed8-48b6-9b45-44146b46a135-istiod-ca-cert\") pod \"68b50881-1ed8-48b6-9b45-44146b46a135\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") "
Oct 30 19:46:52.329864 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.329816    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-kube-api-access-7wsvh\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-kube-api-access-7wsvh\") pod \"68b50881-1ed8-48b6-9b45-44146b46a135\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") "
Oct 30 19:46:52.330041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.329870    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-credential-socket\") pod \"68b50881-1ed8-48b6-9b45-44146b46a135\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") "
Oct 30 19:46:52.330041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.329900    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-socket\") pod \"68b50881-1ed8-48b6-9b45-44146b46a135\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") "
Oct 30 19:46:52.330041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.329906    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-certs" pod "68b50881-1ed8-48b6-9b45-44146b46a135" (UID: "68b50881-1ed8-48b6-9b45-44146b46a135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.330190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.330078    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-data\") pod \"68b50881-1ed8-48b6-9b45-44146b46a135\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") "
Oct 30 19:46:52.330190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.330137    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68b50881-1ed8-48b6-9b45-44146b46a135-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b50881-1ed8-48b6-9b45-44146b46a135-istio-podinfo\") pod \"68b50881-1ed8-48b6-9b45-44146b46a135\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") "
Oct 30 19:46:52.330190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.330158    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-credential-socket" pod "68b50881-1ed8-48b6-9b45-44146b46a135" (UID: "68b50881-1ed8-48b6-9b45-44146b46a135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.330352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.330186    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-istio-token\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-istio-token\") pod \"68b50881-1ed8-48b6-9b45-44146b46a135\" (UID: \"68b50881-1ed8-48b6-9b45-44146b46a135\") "
Oct 30 19:46:52.330352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.330225    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-socket" pod "68b50881-1ed8-48b6-9b45-44146b46a135" (UID: "68b50881-1ed8-48b6-9b45-44146b46a135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.330352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.330275    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b50881-1ed8-48b6-9b45-44146b46a135-istiod-ca-cert" pod "68b50881-1ed8-48b6-9b45-44146b46a135" (UID: "68b50881-1ed8-48b6-9b45-44146b46a135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:52.330525 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.330404    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-data" pod "68b50881-1ed8-48b6-9b45-44146b46a135" (UID: "68b50881-1ed8-48b6-9b45-44146b46a135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.331061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.331023    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b50881-1ed8-48b6-9b45-44146b46a135-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.331252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.331228    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.331372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.331254    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.331372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.331268    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.331372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.331279    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.332373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.332350    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-kube-api-access-7wsvh" pod "68b50881-1ed8-48b6-9b45-44146b46a135" (UID: "68b50881-1ed8-48b6-9b45-44146b46a135"). InnerVolumeSpecName "kube-api-access-7wsvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.332456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.332415    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-envoy" pod "68b50881-1ed8-48b6-9b45-44146b46a135" (UID: "68b50881-1ed8-48b6-9b45-44146b46a135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:52.332965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.332949    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68b50881-1ed8-48b6-9b45-44146b46a135-istio-podinfo" pod "68b50881-1ed8-48b6-9b45-44146b46a135" (UID: "68b50881-1ed8-48b6-9b45-44146b46a135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:52.333083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.333061    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-istio-token" pod "68b50881-1ed8-48b6-9b45-44146b46a135" (UID: "68b50881-1ed8-48b6-9b45-44146b46a135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.383853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.383827    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"]
Oct 30 19:46:52.392495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.392463    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-l8497"]
Oct 30 19:46:52.433242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.433211    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wsvh\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-kube-api-access-7wsvh\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.433393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.433257    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b50881-1ed8-48b6-9b45-44146b46a135-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.433393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.433278    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b50881-1ed8-48b6-9b45-44146b46a135-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.433393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.433295    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b50881-1ed8-48b6-9b45-44146b46a135-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.467827 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.467790    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl" podUID="d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8" containerName="payload-processing" containerID="cri-o://1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0" gracePeriod=25
Oct 30 19:46:52.481450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.481425    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:52.481877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.481851    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:52.482580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.482559    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:46:52.493609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.493582    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl" podUID="db6365b5-d077-49a0-bbd9-100ecada9d1b" containerName="payload-pre-processing" containerID="cri-o://3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f" gracePeriod=25
Oct 30 19:46:52.559980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.559951    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:52.560951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.560921    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:52.563203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.563178    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:46:52.912866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.912837    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:52.925474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.925447    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:52.963804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.963768    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-plugins-config-volume\") pod \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\" (UID: \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\") "
Oct 30 19:46:52.963978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.963844    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db6365b5-d077-49a0-bbd9-100ecada9d1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6365b5-d077-49a0-bbd9-100ecada9d1b-plugins-config-volume\") pod \"db6365b5-d077-49a0-bbd9-100ecada9d1b\" (UID: \"db6365b5-d077-49a0-bbd9-100ecada9d1b\") "
Oct 30 19:46:52.963978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.963904    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-kube-api-access-2c2mm\") pod \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\" (UID: \"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8\") "
Oct 30 19:46:52.964100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.964004    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6365b5-d077-49a0-bbd9-100ecada9d1b-kube-api-access-pcc7v\" (UniqueName: \"kubernetes.io/projected/db6365b5-d077-49a0-bbd9-100ecada9d1b-kube-api-access-pcc7v\") pod \"db6365b5-d077-49a0-bbd9-100ecada9d1b\" (UID: \"db6365b5-d077-49a0-bbd9-100ecada9d1b\") "
Oct 30 19:46:52.965000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.964963    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6365b5-d077-49a0-bbd9-100ecada9d1b-plugins-config-volume" pod "db6365b5-d077-49a0-bbd9-100ecada9d1b" (UID: "db6365b5-d077-49a0-bbd9-100ecada9d1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:52.965210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.965182    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6365b5-d077-49a0-bbd9-100ecada9d1b-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:52.965446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.965420    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-plugins-config-volume" pod "d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8" (UID: "d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:52.967078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.967051    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6365b5-d077-49a0-bbd9-100ecada9d1b-kube-api-access-pcc7v" pod "db6365b5-d077-49a0-bbd9-100ecada9d1b" (UID: "db6365b5-d077-49a0-bbd9-100ecada9d1b"). InnerVolumeSpecName "kube-api-access-pcc7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:52.967453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:52.967430    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-kube-api-access-2c2mm" pod "d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8" (UID: "d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8"). InnerVolumeSpecName "kube-api-access-2c2mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:53.020467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.020405    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:53.056809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.056763    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x" event={"ID":"68b50881-1ed8-48b6-9b45-44146b46a135","Type":"ContainerDied","Data":"1758e20c3a9dc347d59e5d1e85ae3d2b8e47c909a4e0e778839993d80bc7d27f"}
Oct 30 19:46:53.056809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.056812    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"
Oct 30 19:46:53.057069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.056837    4925 scope.go:122] "RemoveContainer" containerID="84ac7c5cd4020f1ac6cc7949cda813948e5b9bda3c221a88839026740b721307"
Oct 30 19:46:53.058681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.058482    4925 generic.go:356] "Generic (PLEG): container finished" podID="d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8" containerID="1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0" exitCode=0
Oct 30 19:46:53.058681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.058513    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"
Oct 30 19:46:53.058681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.058580    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl" event={"ID":"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8","Type":"ContainerDied","Data":"1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0"}
Oct 30 19:46:53.058681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.058640    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl" event={"ID":"d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8","Type":"ContainerDied","Data":"d7f02b559ef41681173fa99861ad1295ee3d7a0c2c5d3f6a22c7e841d23ba501"}
Oct 30 19:46:53.060167 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.060141    4925 generic.go:356] "Generic (PLEG): container finished" podID="db6365b5-d077-49a0-bbd9-100ecada9d1b" containerID="3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f" exitCode=0
Oct 30 19:46:53.060346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.060325    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"
Oct 30 19:46:53.060456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.060191    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl" event={"ID":"db6365b5-d077-49a0-bbd9-100ecada9d1b","Type":"ContainerDied","Data":"3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f"}
Oct 30 19:46:53.060456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.060398    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl" event={"ID":"db6365b5-d077-49a0-bbd9-100ecada9d1b","Type":"ContainerDied","Data":"ac3c23b4f6dd326dda9db70466686737c181227d378814bde0ba23514d971b1e"}
Oct 30 19:46:53.066472 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.066443    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:53.066616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.066597    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8-kube-api-access-2c2mm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:53.066757 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.066621    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcc7v\" (UniqueName: \"kubernetes.io/projected/db6365b5-d077-49a0-bbd9-100ecada9d1b-kube-api-access-pcc7v\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:53.076318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.076302    4925 scope.go:122] "RemoveContainer" containerID="1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0"
Oct 30 19:46:53.091594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.089389    4925 scope.go:122] "RemoveContainer" containerID="1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0"
Oct 30 19:46:53.094398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.094349    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"]
Oct 30 19:46:53.095238 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:46:53.095203    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0\": container with ID starting with 1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0 not found: ID does not exist" containerID="1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0"
Oct 30 19:46:53.095352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.095250    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0"} err="failed to get container status \"1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0\": rpc error: code = NotFound desc = could not find container \"1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0\": container with ID starting with 1fdf0cba355ac7630149c1ed1510da2268d56423466f4a329c8da401a020f0a0 not found: ID does not exist"
Oct 30 19:46:53.095352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.095273    4925 scope.go:122] "RemoveContainer" containerID="3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f"
Oct 30 19:46:53.099553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.099532    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e7123-openshift-default-796568db5f-q2k9x"]
Oct 30 19:46:53.107313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.107255    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"]
Oct 30 19:46:53.108857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.108834    4925 scope.go:122] "RemoveContainer" containerID="3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f"
Oct 30 19:46:53.109206 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:46:53.109185    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f\": container with ID starting with 3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f not found: ID does not exist" containerID="3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f"
Oct 30 19:46:53.109276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.109213    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f"} err="failed to get container status \"3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f\": rpc error: code = NotFound desc = could not find container \"3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f\": container with ID starting with 3aa9c20e3ad4735335442f2b2c99730c692319dc0c7756aa976a16276e535a9f not found: ID does not exist"
Oct 30 19:46:53.111368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.111349    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0a27e-7d57b54c98-j2hcl"]
Oct 30 19:46:53.121088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.121058    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"]
Oct 30 19:46:53.126163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.126140    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0a27e-c76cbb78c-vtdcl"]
Oct 30 19:46:53.263938 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.263914    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" path="/var/lib/kubelet/pods/68b50881-1ed8-48b6-9b45-44146b46a135/volumes"
Oct 30 19:46:53.264471 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.264457    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" path="/var/lib/kubelet/pods/cba538c7-458e-42de-84b0-54d9e11fdcd7/volumes"
Oct 30 19:46:53.264879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.264866    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8" path="/var/lib/kubelet/pods/d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8/volumes"
Oct 30 19:46:53.265562 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.265547    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db6365b5-d077-49a0-bbd9-100ecada9d1b" path="/var/lib/kubelet/pods/db6365b5-d077-49a0-bbd9-100ecada9d1b/volumes"
Oct 30 19:46:53.824192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.824162    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:53.996386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:53.996303    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:54.211915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:54.211877    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"]
Oct 30 19:46:54.215939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:54.215912    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cfa4bb-sj572"]
Oct 30 19:46:54.239318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:54.239283    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:54.492458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:54.492364    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:54.876254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:54.876149    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:55.266654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:55.266622    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c682eed-77b7-4a71-880d-4ee1f0e5b74e" path="/var/lib/kubelet/pods/8c682eed-77b7-4a71-880d-4ee1f0e5b74e/volumes"
Oct 30 19:46:55.681808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:55.681772    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:56.408058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:56.408017    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:56.957184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:56.957155    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:57.038233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038198    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"]
Oct 30 19:46:57.038915 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038899    4925 cpu_manager.go:414] "containerMap: removing container" podUID="83ac2c85-ed1e-448e-b75d-99e9b145c8bc" containerName="revoke-keys"
Oct 30 19:46:57.038977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038917    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ac2c85-ed1e-448e-b75d-99e9b145c8bc" containerName="revoke-keys"
Oct 30 19:46:57.038977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038929    4925 cpu_manager.go:414] "containerMap: removing container" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy"
Oct 30 19:46:57.038977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038934    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy"
Oct 30 19:46:57.038977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038949    4925 cpu_manager.go:414] "containerMap: removing container" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy"
Oct 30 19:46:57.038977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038956    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy"
Oct 30 19:46:57.038977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038962    4925 cpu_manager.go:414] "containerMap: removing container" podUID="db6365b5-d077-49a0-bbd9-100ecada9d1b" containerName="payload-pre-processing"
Oct 30 19:46:57.038977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038967    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db6365b5-d077-49a0-bbd9-100ecada9d1b" containerName="payload-pre-processing"
Oct 30 19:46:57.039206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038994    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8" containerName="payload-processing"
Oct 30 19:46:57.039206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.038999    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8" containerName="payload-processing"
Oct 30 19:46:57.039206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.039103    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy"
Oct 30 19:46:57.039206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.039112    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="db6365b5-d077-49a0-bbd9-100ecada9d1b" containerName="payload-pre-processing"
Oct 30 19:46:57.039206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.039122    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f42eb6-90e6-403a-b6b4-04ab67b8fdf8" containerName="payload-processing"
Oct 30 19:46:57.039206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.039132    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ac2c85-ed1e-448e-b75d-99e9b145c8bc" containerName="revoke-keys"
Oct 30 19:46:57.039206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.039139    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba538c7-458e-42de-84b0-54d9e11fdcd7" containerName="istio-proxy"
Oct 30 19:46:57.045143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.045122    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.046261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.046232    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"]
Oct 30 19:46:57.052661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.052041    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:46:57.052661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.052113    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:57.054052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.054033    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"]
Oct 30 19:46:57.083524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.061761    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"]
Oct 30 19:46:57.083524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.062823    4925 cpu_manager.go:414] "containerMap: removing container" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy"
Oct 30 19:46:57.083524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.062848    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy"
Oct 30 19:46:57.083524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.063105    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b50881-1ed8-48b6-9b45-44146b46a135" containerName="istio-proxy"
Oct 30 19:46:57.083524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.074478    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9a48d5-vdx9q"]
Oct 30 19:46:57.083524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.074542    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"]
Oct 30 19:46:57.083524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.074721    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.349774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.342218    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:57.383950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.383917    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428eb372-ec1f-4905-84e6-983bfd0cc5c6" path="/var/lib/kubelet/pods/428eb372-ec1f-4905-84e6-983bfd0cc5c6/volumes"
Oct 30 19:46:57.433854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.433823    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-278885ed-8x2t7\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.434234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.433940    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvd9\" (UniqueName: \"kubernetes.io/projected/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-kube-api-access-5jvd9\") pod \"maas-api-revoke-sub-f75c6cae-7zhnf\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.434234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.433966    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqz2\" (UniqueName: \"kubernetes.io/projected/84cb409f-a07b-42d7-b58a-9913245b09a3-kube-api-access-dfqz2\") pod \"maas-api-revoke-sub-278885ed-8x2t7\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.434234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.434093    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f75c6cae-7zhnf\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.434234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.434143    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-278885ed-8x2t7\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.434389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.434300    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f75c6cae-7zhnf\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.562775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.561273    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-278885ed-8x2t7\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.562775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.561407    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f75c6cae-7zhnf\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.562775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.561606    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-278885ed-8x2t7\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.562775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.561721    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvd9\" (UniqueName: \"kubernetes.io/projected/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-kube-api-access-5jvd9\") pod \"maas-api-revoke-sub-f75c6cae-7zhnf\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.562775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.561767    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqz2\" (UniqueName: \"kubernetes.io/projected/84cb409f-a07b-42d7-b58a-9913245b09a3-kube-api-access-dfqz2\") pod \"maas-api-revoke-sub-278885ed-8x2t7\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.562775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.562010    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f75c6cae-7zhnf\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.564393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.564212    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-278885ed-8x2t7\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.570181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.570135    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f75c6cae-7zhnf\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.597178 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.597145    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-278885ed-8x2t7\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.597398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.597145    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f75c6cae-7zhnf\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.603773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.603637    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:57.606850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.606819    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqz2\" (UniqueName: \"kubernetes.io/projected/84cb409f-a07b-42d7-b58a-9913245b09a3-kube-api-access-dfqz2\") pod \"maas-api-revoke-sub-278885ed-8x2t7\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.608179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.608155    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvd9\" (UniqueName: \"kubernetes.io/projected/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-kube-api-access-5jvd9\") pod \"maas-api-revoke-sub-f75c6cae-7zhnf\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.679322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.679288    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:46:57.686134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.686111    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:46:57.967814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.967786    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"]
Oct 30 19:46:57.968300 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:57.968277    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5fa4b3_aa17_418a_a826_2fe9c22dfc50.slice/crio-e46739c569a05c859729474209120b4e026ca8f0bdbb25d786e8272929663e7b WatchSource:0}: Error finding container e46739c569a05c859729474209120b4e026ca8f0bdbb25d786e8272929663e7b: Status 404 returned error can't find the container with id e46739c569a05c859729474209120b4e026ca8f0bdbb25d786e8272929663e7b
Oct 30 19:46:57.989653 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:46:57.989628    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cb409f_a07b_42d7_b58a_9913245b09a3.slice/crio-85214121c46a7d26d7fd54e51a0decb62524bad9fc8767a1ac7d2868725ee6f5 WatchSource:0}: Error finding container 85214121c46a7d26d7fd54e51a0decb62524bad9fc8767a1ac7d2868725ee6f5: Status 404 returned error can't find the container with id 85214121c46a7d26d7fd54e51a0decb62524bad9fc8767a1ac7d2868725ee6f5
Oct 30 19:46:57.989968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:57.989946    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"]
Oct 30 19:46:58.294889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:58.294668    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:58.382764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:58.382725    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7" event={"ID":"84cb409f-a07b-42d7-b58a-9913245b09a3","Type":"ContainerStarted","Data":"85214121c46a7d26d7fd54e51a0decb62524bad9fc8767a1ac7d2868725ee6f5"}
Oct 30 19:46:58.384166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:58.384138    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf" event={"ID":"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50","Type":"ContainerStarted","Data":"e46739c569a05c859729474209120b4e026ca8f0bdbb25d786e8272929663e7b"}
Oct 30 19:46:58.459985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:58.459953    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:59.036858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.030903    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:59.215372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.215227    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:59.393771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.386203    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:59.501710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.498739    4925 generic.go:356] "Generic (PLEG): container finished" podID="ff5fa4b3-aa17-418a-a826-2fe9c22dfc50" containerID="a8c63f0e754eb67917282b21c02df09a201e8dcb90db3f36e5e7fe8b727d386a" exitCode=0
Oct 30 19:46:59.501710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.498983    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf" event={"ID":"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50","Type":"ContainerDied","Data":"a8c63f0e754eb67917282b21c02df09a201e8dcb90db3f36e5e7fe8b727d386a"}
Oct 30 19:46:59.521753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.502408    4925 generic.go:356] "Generic (PLEG): container finished" podID="84cb409f-a07b-42d7-b58a-9913245b09a3" containerID="88e9a1f220e4cd25facb593b07a6948738c444b681dbdd854b3ef3067de4ef18" exitCode=0
Oct 30 19:46:59.521753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.502571    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7" event={"ID":"84cb409f-a07b-42d7-b58a-9913245b09a3","Type":"ContainerDied","Data":"88e9a1f220e4cd25facb593b07a6948738c444b681dbdd854b3ef3067de4ef18"}
Oct 30 19:46:59.741849 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.741754    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"]
Oct 30 19:46:59.747051 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.747009    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a844e-bgvmr"]
Oct 30 19:46:59.805673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:46:59.805637    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:00.083743 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.082931    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:00.294038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.294005    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:00.694279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.694238    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:47:00.746603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.746575    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:47:00.757552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.757524    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-openshift-ca\") pod \"84cb409f-a07b-42d7-b58a-9913245b09a3\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") "
Oct 30 19:47:00.757810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.757789    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-xks-ca\") pod \"84cb409f-a07b-42d7-b58a-9913245b09a3\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") "
Oct 30 19:47:00.757942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.757899    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84cb409f-a07b-42d7-b58a-9913245b09a3-kube-api-access-dfqz2\" (UniqueName: \"kubernetes.io/projected/84cb409f-a07b-42d7-b58a-9913245b09a3-kube-api-access-dfqz2\") pod \"84cb409f-a07b-42d7-b58a-9913245b09a3\" (UID: \"84cb409f-a07b-42d7-b58a-9913245b09a3\") "
Oct 30 19:47:00.758018 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.757936    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-openshift-ca" pod "84cb409f-a07b-42d7-b58a-9913245b09a3" (UID: "84cb409f-a07b-42d7-b58a-9913245b09a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:00.758845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.758819    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:00.760259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.760229    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-xks-ca" pod "84cb409f-a07b-42d7-b58a-9913245b09a3" (UID: "84cb409f-a07b-42d7-b58a-9913245b09a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:00.760421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.760397    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84cb409f-a07b-42d7-b58a-9913245b09a3-kube-api-access-dfqz2" pod "84cb409f-a07b-42d7-b58a-9913245b09a3" (UID: "84cb409f-a07b-42d7-b58a-9913245b09a3"). InnerVolumeSpecName "kube-api-access-dfqz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:00.859537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.859507    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-xks-ca\") pod \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") "
Oct 30 19:47:00.859744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.859551    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-openshift-ca\") pod \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") "
Oct 30 19:47:00.859857 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.859831    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-kube-api-access-5jvd9\" (UniqueName: \"kubernetes.io/projected/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-kube-api-access-5jvd9\") pod \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\" (UID: \"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50\") "
Oct 30 19:47:00.859997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.859977    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-openshift-ca" pod "ff5fa4b3-aa17-418a-a826-2fe9c22dfc50" (UID: "ff5fa4b3-aa17-418a-a826-2fe9c22dfc50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:00.860527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.860508    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:00.860576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.860531    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cb409f-a07b-42d7-b58a-9913245b09a3-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:00.860576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.860541    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqz2\" (UniqueName: \"kubernetes.io/projected/84cb409f-a07b-42d7-b58a-9913245b09a3-kube-api-access-dfqz2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:00.862004 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.861981    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-xks-ca" pod "ff5fa4b3-aa17-418a-a826-2fe9c22dfc50" (UID: "ff5fa4b3-aa17-418a-a826-2fe9c22dfc50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:00.862279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.862252    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-kube-api-access-5jvd9" pod "ff5fa4b3-aa17-418a-a826-2fe9c22dfc50" (UID: "ff5fa4b3-aa17-418a-a826-2fe9c22dfc50"). InnerVolumeSpecName "kube-api-access-5jvd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:00.962199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.962169    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvd9\" (UniqueName: \"kubernetes.io/projected/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-kube-api-access-5jvd9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:00.962199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:00.962196    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:01.203277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:01.202922    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:01.264551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:01.264516    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51253e1f-197c-4ad8-951f-0a93d314424e" path="/var/lib/kubelet/pods/51253e1f-197c-4ad8-951f-0a93d314424e/volumes"
Oct 30 19:47:01.520766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:01.520677    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"
Oct 30 19:47:01.520766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:01.520715    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf" event={"ID":"ff5fa4b3-aa17-418a-a826-2fe9c22dfc50","Type":"ContainerDied","Data":"e46739c569a05c859729474209120b4e026ca8f0bdbb25d786e8272929663e7b"}
Oct 30 19:47:01.520766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:01.520765    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e46739c569a05c859729474209120b4e026ca8f0bdbb25d786e8272929663e7b"
Oct 30 19:47:01.522267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:01.522236    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"
Oct 30 19:47:01.522267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:01.522240    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7" event={"ID":"84cb409f-a07b-42d7-b58a-9913245b09a3","Type":"ContainerDied","Data":"85214121c46a7d26d7fd54e51a0decb62524bad9fc8767a1ac7d2868725ee6f5"}
Oct 30 19:47:01.522431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:01.522284    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85214121c46a7d26d7fd54e51a0decb62524bad9fc8767a1ac7d2868725ee6f5"
Oct 30 19:47:01.959220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:01.959181    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:02.579124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:02.579082    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:02.742565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:02.742530    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:02.860345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:02.860243    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"]
Oct 30 19:47:02.860539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:02.860488    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy" containerID="cri-o://2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740" gracePeriod=30
Oct 30 19:47:02.918711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:02.918658    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:03.107359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:03.107323    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:04.098891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:04.098816    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:04.620649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:04.620610    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:05.067713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:05.067390    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:05.380264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:05.380189    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:06.475715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:06.475029    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:06.647724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:06.647645    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main" containerID="cri-o://bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d" gracePeriod=45
Oct 30 19:47:06.677812 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:06.677759    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main" containerID="cri-o://be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2" gracePeriod=45
Oct 30 19:47:06.973546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:06.973516    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:47:07.005618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.005590    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:47:07.029183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.029122    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"]
Oct 30 19:47:07.036869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.035835    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affab8b4-2jjhw"]
Oct 30 19:47:07.137089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137061    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-dshm\") pod \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") "
Oct 30 19:47:07.137285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137102    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-dshm\") pod \"d4e73920-396b-4a7d-9573-a9009d5f5888\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") "
Oct 30 19:47:07.137285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137142    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-model-cache\") pod \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") "
Oct 30 19:47:07.137285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137179    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-kube-api-access-p69cn\" (UniqueName: \"kubernetes.io/projected/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-kube-api-access-p69cn\") pod \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") "
Oct 30 19:47:07.137285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137237    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4e73920-396b-4a7d-9573-a9009d5f5888-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e73920-396b-4a7d-9573-a9009d5f5888-tls-certs\") pod \"d4e73920-396b-4a7d-9573-a9009d5f5888\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") "
Oct 30 19:47:07.137527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137304    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tmp-dir\") pod \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") "
Oct 30 19:47:07.137527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137323    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tls-certs\") pod \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") "
Oct 30 19:47:07.137527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137401    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-model-cache\") pod \"d4e73920-396b-4a7d-9573-a9009d5f5888\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") "
Oct 30 19:47:07.137527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137425    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-model-cache" pod "44c77b77-cb30-4abf-ba5a-1747dae4d2e1" (UID: "44c77b77-cb30-4abf-ba5a-1747dae4d2e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.137527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137481    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-home\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-home\") pod \"d4e73920-396b-4a7d-9573-a9009d5f5888\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") "
Oct 30 19:47:07.137859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137630    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-tmp-dir\") pod \"d4e73920-396b-4a7d-9573-a9009d5f5888\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") "
Oct 30 19:47:07.137859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137661    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e73920-396b-4a7d-9573-a9009d5f5888-kube-api-access-lpkhq\" (UniqueName: \"kubernetes.io/projected/d4e73920-396b-4a7d-9573-a9009d5f5888-kube-api-access-lpkhq\") pod \"d4e73920-396b-4a7d-9573-a9009d5f5888\" (UID: \"d4e73920-396b-4a7d-9573-a9009d5f5888\") "
Oct 30 19:47:07.137859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137746    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-model-cache" pod "d4e73920-396b-4a7d-9573-a9009d5f5888" (UID: "d4e73920-396b-4a7d-9573-a9009d5f5888"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.137859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137809    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-home\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-home\") pod \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\" (UID: \"44c77b77-cb30-4abf-ba5a-1747dae4d2e1\") "
Oct 30 19:47:07.138089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.137937    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tmp-dir" pod "44c77b77-cb30-4abf-ba5a-1747dae4d2e1" (UID: "44c77b77-cb30-4abf-ba5a-1747dae4d2e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.138089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.138106    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-home" pod "d4e73920-396b-4a7d-9573-a9009d5f5888" (UID: "d4e73920-396b-4a7d-9573-a9009d5f5888"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.138434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.138370    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-tmp-dir" pod "d4e73920-396b-4a7d-9573-a9009d5f5888" (UID: "d4e73920-396b-4a7d-9573-a9009d5f5888"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.138568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.138549    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-home" pod "44c77b77-cb30-4abf-ba5a-1747dae4d2e1" (UID: "44c77b77-cb30-4abf-ba5a-1747dae4d2e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.138837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.138746    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.138837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.138769    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.138837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.138783    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.138837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.138795    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.139120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.138810    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.139120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.138865    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.140126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.140095    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tls-certs" pod "44c77b77-cb30-4abf-ba5a-1747dae4d2e1" (UID: "44c77b77-cb30-4abf-ba5a-1747dae4d2e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:07.140326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.140301    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-dshm" pod "44c77b77-cb30-4abf-ba5a-1747dae4d2e1" (UID: "44c77b77-cb30-4abf-ba5a-1747dae4d2e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.140379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.140358    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-kube-api-access-p69cn" pod "44c77b77-cb30-4abf-ba5a-1747dae4d2e1" (UID: "44c77b77-cb30-4abf-ba5a-1747dae4d2e1"). InnerVolumeSpecName "kube-api-access-p69cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:07.140424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.140408    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-dshm" pod "d4e73920-396b-4a7d-9573-a9009d5f5888" (UID: "d4e73920-396b-4a7d-9573-a9009d5f5888"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:07.140479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.140464    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e73920-396b-4a7d-9573-a9009d5f5888-tls-certs" pod "d4e73920-396b-4a7d-9573-a9009d5f5888" (UID: "d4e73920-396b-4a7d-9573-a9009d5f5888"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:07.140584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.140567    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e73920-396b-4a7d-9573-a9009d5f5888-kube-api-access-lpkhq" pod "d4e73920-396b-4a7d-9573-a9009d5f5888" (UID: "d4e73920-396b-4a7d-9573-a9009d5f5888"). InnerVolumeSpecName "kube-api-access-lpkhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:07.240469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.240379    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpkhq\" (UniqueName: \"kubernetes.io/projected/d4e73920-396b-4a7d-9573-a9009d5f5888-kube-api-access-lpkhq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.240469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.240408    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.240469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.240418    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e73920-396b-4a7d-9573-a9009d5f5888-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.240469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.240426    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p69cn\" (UniqueName: \"kubernetes.io/projected/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-kube-api-access-p69cn\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.240469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.240436    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e73920-396b-4a7d-9573-a9009d5f5888-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.240469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.240444    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c77b77-cb30-4abf-ba5a-1747dae4d2e1-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:07.265283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.265256    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c79bd8-12e7-4c04-9774-e2c638b2548f" path="/var/lib/kubelet/pods/93c79bd8-12e7-4c04-9774-e2c638b2548f/volumes"
Oct 30 19:47:07.506779 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.506285    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:07.611254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.611218    4925 generic.go:356] "Generic (PLEG): container finished" podID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerID="be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2" exitCode=0
Oct 30 19:47:07.611441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.611258    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"
Oct 30 19:47:07.611441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.611281    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" event={"ID":"44c77b77-cb30-4abf-ba5a-1747dae4d2e1","Type":"ContainerDied","Data":"be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2"}
Oct 30 19:47:07.611441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.611337    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk" event={"ID":"44c77b77-cb30-4abf-ba5a-1747dae4d2e1","Type":"ContainerDied","Data":"84cb75cb778032761dc4c00e88dce16366102fd602bb544488dddea11bc32c16"}
Oct 30 19:47:07.611441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.611359    4925 scope.go:122] "RemoveContainer" containerID="be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2"
Oct 30 19:47:07.613469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.613446    4925 generic.go:356] "Generic (PLEG): container finished" podID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerID="bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d" exitCode=0
Oct 30 19:47:07.613840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.613488    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"
Oct 30 19:47:07.613840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.613527    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" event={"ID":"d4e73920-396b-4a7d-9573-a9009d5f5888","Type":"ContainerDied","Data":"bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d"}
Oct 30 19:47:07.613840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.613564    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb" event={"ID":"d4e73920-396b-4a7d-9573-a9009d5f5888","Type":"ContainerDied","Data":"73bd250cebf7cb9c9d7ee917a11f1b3682e8de9f8a7ad1a7b22de4057f1421e5"}
Oct 30 19:47:07.624526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.624510    4925 scope.go:122] "RemoveContainer" containerID="be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2"
Oct 30 19:47:07.624917 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:07.624896    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2\": container with ID starting with be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2 not found: ID does not exist" containerID="be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2"
Oct 30 19:47:07.625017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.624926    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2"} err="failed to get container status \"be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2\": rpc error: code = NotFound desc = could not find container \"be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2\": container with ID starting with be2fe2afe9229fd3078d3ef70b6a0008825362abca7cc891d7fc88e94493b4d2 not found: ID does not exist"
Oct 30 19:47:07.625017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.624947    4925 scope.go:122] "RemoveContainer" containerID="bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d"
Oct 30 19:47:07.635392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.635373    4925 scope.go:122] "RemoveContainer" containerID="bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d"
Oct 30 19:47:07.635701 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:07.635669    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d\": container with ID starting with bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d not found: ID does not exist" containerID="bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d"
Oct 30 19:47:07.635769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.635713    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d"} err="failed to get container status \"bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d\": rpc error: code = NotFound desc = could not find container \"bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d\": container with ID starting with bdee450cb402970c4ef2e1d460b5d567f840954780e0b56f1bc18297d262d68d not found: ID does not exist"
Oct 30 19:47:07.663540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.663505    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"]
Oct 30 19:47:07.682783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.682735    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/distinct-w0-ad23c7-kserve-78dd6565c7-lvrqb"]
Oct 30 19:47:07.728038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.728001    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"]
Oct 30 19:47:07.728337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.728316    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ad23c7/simulator-w0-ad23c7-kserve-576499569f-bkjmk"]
Oct 30 19:47:07.752755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.752712    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"]
Oct 30 19:47:07.753043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.752978    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy" containerID="cri-o://f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf" gracePeriod=30
Oct 30 19:47:07.763518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.763455    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"]
Oct 30 19:47:07.763738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.763679    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy" containerID="cri-o://e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91" gracePeriod=30
Oct 30 19:47:07.776918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.776893    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"]
Oct 30 19:47:07.777112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.777087    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy" containerID="cri-o://26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32" gracePeriod=30
Oct 30 19:47:07.779269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.779240    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:07.974964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:07.971968    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.006119    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"]
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007077    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007096    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007123    4925 cpu_manager.go:414] "containerMap: removing container" podUID="84cb409f-a07b-42d7-b58a-9913245b09a3" containerName="revoke-keys"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007130    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84cb409f-a07b-42d7-b58a-9913245b09a3" containerName="revoke-keys"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007140    4925 cpu_manager.go:414] "containerMap: removing container" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007147    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007213    4925 cpu_manager.go:414] "containerMap: removing container" podUID="ff5fa4b3-aa17-418a-a826-2fe9c22dfc50" containerName="revoke-keys"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007221    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5fa4b3-aa17-418a-a826-2fe9c22dfc50" containerName="revoke-keys"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007358    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5fa4b3-aa17-418a-a826-2fe9c22dfc50" containerName="revoke-keys"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007372    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" containerName="main"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007383    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="84cb409f-a07b-42d7-b58a-9913245b09a3" containerName="revoke-keys"
Oct 30 19:47:08.007714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.007392    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" containerName="main"
Oct 30 19:47:08.074680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.073877    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"]
Oct 30 19:47:08.074680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.073999    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.086280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.085941    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:08.086280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.086095    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:47:08.140134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.140092    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ln7z\" (UniqueName: \"kubernetes.io/projected/8bd50f91-9c64-459c-a604-5cbfff4dedbf-kube-api-access-7ln7z\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.140315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.140210    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.140458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.140441    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.242233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.242196    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.242427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.242304    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ln7z\" (UniqueName: \"kubernetes.io/projected/8bd50f91-9c64-459c-a604-5cbfff4dedbf-kube-api-access-7ln7z\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.242427 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.242388    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.243219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.243193    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.245496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.245467    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.256229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.256205    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ln7z\" (UniqueName: \"kubernetes.io/projected/8bd50f91-9c64-459c-a604-5cbfff4dedbf-kube-api-access-7ln7z\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.282384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.282324    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"]
Oct 30 19:47:08.303985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.303958    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"]
Oct 30 19:47:08.304151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.304089    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.348510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.348485    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:47:08.417347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.417311    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:08.444832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.444803    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-kube-api-access-m52lq\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-kube-api-access-m52lq\") pod \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") "
Oct 30 19:47:08.444976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.444865    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-token\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-token\") pod \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") "
Oct 30 19:47:08.444976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.444911    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-envoy\") pod \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") "
Oct 30 19:47:08.445092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445035    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istiod-ca-cert\") pod \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") "
Oct 30 19:47:08.445148 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445088    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-credential-socket\") pod \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") "
Oct 30 19:47:08.445223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445209    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-socket\") pod \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") "
Oct 30 19:47:08.445589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445386    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-podinfo\") pod \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") "
Oct 30 19:47:08.445589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445430    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-certs\") pod \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") "
Oct 30 19:47:08.445589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445489    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-data\") pod \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\" (UID: \"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02\") "
Oct 30 19:47:08.445589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445536    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-credential-socket" pod "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" (UID: "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.445589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445555    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istiod-ca-cert" pod "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" (UID: "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445896    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q7sv\" (UniqueName: \"kubernetes.io/projected/29a69548-f46b-4c04-a93f-01ba17a0fa07-kube-api-access-8q7sv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.445998    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-certs" pod "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" (UID: "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.446019    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.446141    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-data" pod "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" (UID: "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.446174    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-socket" pod "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" (UID: "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.446311    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.446410    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.446423    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.446436    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.446450    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.446551 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.446458    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.447394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.447370    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-token" pod "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" (UID: "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:08.447916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.447888    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-kube-api-access-m52lq" pod "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" (UID: "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02"). InnerVolumeSpecName "kube-api-access-m52lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:08.448506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.448483    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-envoy" pod "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" (UID: "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:08.449097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.449073    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-podinfo" pod "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" (UID: "d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:08.548073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.548036    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.548567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.548253    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q7sv\" (UniqueName: \"kubernetes.io/projected/29a69548-f46b-4c04-a93f-01ba17a0fa07-kube-api-access-8q7sv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.548567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.548362    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.548567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.548485    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.548567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.548502    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m52lq\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-kube-api-access-m52lq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.548567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.548516    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.548567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.548530    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:08.548945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.548921    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.551104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.551078    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.560067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.560045    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q7sv\" (UniqueName: \"kubernetes.io/projected/29a69548-f46b-4c04-a93f-01ba17a0fa07-kube-api-access-8q7sv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.628781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.628670    4925 generic.go:356] "Generic (PLEG): container finished" podID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerID="2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740" exitCode=0
Oct 30 19:47:08.628781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.628743    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"
Oct 30 19:47:08.628781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.628752    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" event={"ID":"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02","Type":"ContainerDied","Data":"2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740"}
Oct 30 19:47:08.629052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.628802    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn" event={"ID":"d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02","Type":"ContainerDied","Data":"f05f2203b36132b290afc812ee4c60c9fd4af347ab8e247343c44eb645e90b47"}
Oct 30 19:47:08.629052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.628823    4925 scope.go:122] "RemoveContainer" containerID="2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740"
Oct 30 19:47:08.645798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.645767    4925 scope.go:122] "RemoveContainer" containerID="2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740"
Oct 30 19:47:08.646343 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:08.646313    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740\": container with ID starting with 2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740 not found: ID does not exist" containerID="2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740"
Oct 30 19:47:08.646509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.646484    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740"} err="failed to get container status \"2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740\": rpc error: code = NotFound desc = could not find container \"2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740\": container with ID starting with 2950ea82352ab7a79867c55810e21199d0e54a60215e8ff2709007a02f14c740 not found: ID does not exist"
Oct 30 19:47:08.646811 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.646788    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:08.665077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.665047    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"]
Oct 30 19:47:08.669882 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.669860    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cfa4bb-openshift-default-75f585bdc-h92cn"]
Oct 30 19:47:08.709710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.709663    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"]
Oct 30 19:47:08.971590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.969288    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"]
Oct 30 19:47:08.985624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:08.985584    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:08.986215 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:47:08.986186    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a69548_f46b_4c04_a93f_01ba17a0fa07.slice/crio-5083f819fe4176bf548951720d9d87b699716ddc0561d43c647df014bc9318b1 WatchSource:0}: Error finding container 5083f819fe4176bf548951720d9d87b699716ddc0561d43c647df014bc9318b1: Status 404 returned error can't find the container with id 5083f819fe4176bf548951720d9d87b699716ddc0561d43c647df014bc9318b1
Oct 30 19:47:09.264438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:09.264369    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c77b77-cb30-4abf-ba5a-1747dae4d2e1" path="/var/lib/kubelet/pods/44c77b77-cb30-4abf-ba5a-1747dae4d2e1/volumes"
Oct 30 19:47:09.264895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:09.264880    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" path="/var/lib/kubelet/pods/d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02/volumes"
Oct 30 19:47:09.265323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:09.265309    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e73920-396b-4a7d-9573-a9009d5f5888" path="/var/lib/kubelet/pods/d4e73920-396b-4a7d-9573-a9009d5f5888/volumes"
Oct 30 19:47:09.639581 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:09.639552    4925 generic.go:356] "Generic (PLEG): container finished" podID="8bd50f91-9c64-459c-a604-5cbfff4dedbf" containerID="f5db5fc996447d831817d4918463509265431e2017da4a46af068b1b6b7cc366" exitCode=0
Oct 30 19:47:09.639997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:09.639588    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp" event={"ID":"8bd50f91-9c64-459c-a604-5cbfff4dedbf","Type":"ContainerDied","Data":"f5db5fc996447d831817d4918463509265431e2017da4a46af068b1b6b7cc366"}
Oct 30 19:47:09.639997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:09.639651    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp" event={"ID":"8bd50f91-9c64-459c-a604-5cbfff4dedbf","Type":"ContainerStarted","Data":"ab8a2410f86d67c392533adfdefd2aed9530e0c2e7a1fcc0eee622425aa32a14"}
Oct 30 19:47:09.642764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:09.642732    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg" event={"ID":"29a69548-f46b-4c04-a93f-01ba17a0fa07","Type":"ContainerStarted","Data":"10b3b9de7c58b08f6607027a9a745b8ada582c65125eb2bf58ec4bd2848ab6df"}
Oct 30 19:47:09.642885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:09.642785    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg" event={"ID":"29a69548-f46b-4c04-a93f-01ba17a0fa07","Type":"ContainerStarted","Data":"5083f819fe4176bf548951720d9d87b699716ddc0561d43c647df014bc9318b1"}
Oct 30 19:47:09.791007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:09.790879    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:10.653861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:10.653825    4925 generic.go:356] "Generic (PLEG): container finished" podID="29a69548-f46b-4c04-a93f-01ba17a0fa07" containerID="10b3b9de7c58b08f6607027a9a745b8ada582c65125eb2bf58ec4bd2848ab6df" exitCode=0
Oct 30 19:47:10.654332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:10.653918    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg" event={"ID":"29a69548-f46b-4c04-a93f-01ba17a0fa07","Type":"ContainerDied","Data":"10b3b9de7c58b08f6607027a9a745b8ada582c65125eb2bf58ec4bd2848ab6df"}
Oct 30 19:47:10.833073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:10.833047    4925 scope.go:122] "RemoveContainer" containerID="866a1a5a6adc755823127fa343a9748f0c2c89bae54a1030eb2490ac1e171a7e"
Oct 30 19:47:10.850186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:10.850157    4925 scope.go:122] "RemoveContainer" containerID="28149fed15278f4aa1730ef0f08c72f73932f94bebdc5223c12d70ad4bdddd5b"
Oct 30 19:47:10.852588 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:10.852566    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:10.905627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:10.905599    4925 scope.go:122] "RemoveContainer" containerID="de1ddca24f53b9b463881c13bf7fd0352c164ae418191425ae7f95983ddd990b"
Oct 30 19:47:10.917869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:10.917843    4925 scope.go:122] "RemoveContainer" containerID="d35ba3d709e2e86847e08dcf3b023d975f86673246494d84d9d800e0e5eeaa72"
Oct 30 19:47:10.931804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:10.931733    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:11.010638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.010613    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-xks-ca\") pod \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") "
Oct 30 19:47:11.010897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.010880    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd50f91-9c64-459c-a604-5cbfff4dedbf-kube-api-access-7ln7z\" (UniqueName: \"kubernetes.io/projected/8bd50f91-9c64-459c-a604-5cbfff4dedbf-kube-api-access-7ln7z\") pod \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") "
Oct 30 19:47:11.011030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.011015    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-openshift-ca\") pod \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\" (UID: \"8bd50f91-9c64-459c-a604-5cbfff4dedbf\") "
Oct 30 19:47:11.011393 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.011325    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-openshift-ca" pod "8bd50f91-9c64-459c-a604-5cbfff4dedbf" (UID: "8bd50f91-9c64-459c-a604-5cbfff4dedbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:11.011869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.011843    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:11.013014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.012990    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd50f91-9c64-459c-a604-5cbfff4dedbf-kube-api-access-7ln7z" pod "8bd50f91-9c64-459c-a604-5cbfff4dedbf" (UID: "8bd50f91-9c64-459c-a604-5cbfff4dedbf"). InnerVolumeSpecName "kube-api-access-7ln7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:11.013215 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.013193    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-xks-ca" pod "8bd50f91-9c64-459c-a604-5cbfff4dedbf" (UID: "8bd50f91-9c64-459c-a604-5cbfff4dedbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:11.113650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.113618    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd50f91-9c64-459c-a604-5cbfff4dedbf-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:11.113650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.113646    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ln7z\" (UniqueName: \"kubernetes.io/projected/8bd50f91-9c64-459c-a604-5cbfff4dedbf-kube-api-access-7ln7z\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:11.664673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.664641    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"
Oct 30 19:47:11.665310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.664673    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp" event={"ID":"8bd50f91-9c64-459c-a604-5cbfff4dedbf","Type":"ContainerDied","Data":"ab8a2410f86d67c392533adfdefd2aed9530e0c2e7a1fcc0eee622425aa32a14"}
Oct 30 19:47:11.665310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.664749    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8a2410f86d67c392533adfdefd2aed9530e0c2e7a1fcc0eee622425aa32a14"
Oct 30 19:47:11.835496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.835469    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:11.923446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.923351    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a69548-f46b-4c04-a93f-01ba17a0fa07-kube-api-access-8q7sv\" (UniqueName: \"kubernetes.io/projected/29a69548-f46b-4c04-a93f-01ba17a0fa07-kube-api-access-8q7sv\") pod \"29a69548-f46b-4c04-a93f-01ba17a0fa07\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") "
Oct 30 19:47:11.923625 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.923536    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-openshift-ca\") pod \"29a69548-f46b-4c04-a93f-01ba17a0fa07\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") "
Oct 30 19:47:11.923821 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.923802    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-xks-ca\") pod \"29a69548-f46b-4c04-a93f-01ba17a0fa07\" (UID: \"29a69548-f46b-4c04-a93f-01ba17a0fa07\") "
Oct 30 19:47:11.923927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.923883    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-openshift-ca" pod "29a69548-f46b-4c04-a93f-01ba17a0fa07" (UID: "29a69548-f46b-4c04-a93f-01ba17a0fa07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:11.924341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.924314    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:11.925814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.925792    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a69548-f46b-4c04-a93f-01ba17a0fa07-kube-api-access-8q7sv" pod "29a69548-f46b-4c04-a93f-01ba17a0fa07" (UID: "29a69548-f46b-4c04-a93f-01ba17a0fa07"). InnerVolumeSpecName "kube-api-access-8q7sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:11.926006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.925987    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-xks-ca" pod "29a69548-f46b-4c04-a93f-01ba17a0fa07" (UID: "29a69548-f46b-4c04-a93f-01ba17a0fa07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:11.993742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:11.992930    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:12.026623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:12.026585    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a69548-f46b-4c04-a93f-01ba17a0fa07-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:12.026623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:12.026626    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q7sv\" (UniqueName: \"kubernetes.io/projected/29a69548-f46b-4c04-a93f-01ba17a0fa07-kube-api-access-8q7sv\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:12.675233 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:12.675203    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"
Oct 30 19:47:12.675655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:12.675243    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg" event={"ID":"29a69548-f46b-4c04-a93f-01ba17a0fa07","Type":"ContainerDied","Data":"5083f819fe4176bf548951720d9d87b699716ddc0561d43c647df014bc9318b1"}
Oct 30 19:47:12.675655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:12.675299    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5083f819fe4176bf548951720d9d87b699716ddc0561d43c647df014bc9318b1"
Oct 30 19:47:13.036996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.034661    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:13.143901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.143434    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:47:13.144078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.143996    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"]
Oct 30 19:47:13.153214 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.153186    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"]
Oct 30 19:47:13.226123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.225912    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:47:13.228837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.228812    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:47:13.260552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.260516    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-data\") pod \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") "
Oct 30 19:47:13.265254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.265228    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-kube-api-access-wfnrl\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-kube-api-access-wfnrl\") pod \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") "
Oct 30 19:47:13.265495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.265474    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-envoy\") pod \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") "
Oct 30 19:47:13.265674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.265655    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494c707f-dedc-4a62-b3fe-62aa26c6c853-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c707f-dedc-4a62-b3fe-62aa26c6c853-istiod-ca-cert\") pod \"494c707f-dedc-4a62-b3fe-62aa26c6c853\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") "
Oct 30 19:47:13.265913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.265893    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca278fd-dae1-47a5-8bd4-b350a4820500-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca278fd-dae1-47a5-8bd4-b350a4820500-istiod-ca-cert\") pod \"2ca278fd-dae1-47a5-8bd4-b350a4820500\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") "
Oct 30 19:47:13.266061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266045    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-data\") pod \"494c707f-dedc-4a62-b3fe-62aa26c6c853\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") "
Oct 30 19:47:13.266229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266211    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-envoy\") pod \"2ca278fd-dae1-47a5-8bd4-b350a4820500\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") "
Oct 30 19:47:13.266412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266395    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-certs\") pod \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") "
Oct 30 19:47:13.266655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266632    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-socket\") pod \"2ca278fd-dae1-47a5-8bd4-b350a4820500\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") "
Oct 30 19:47:13.266770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266681    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-certs\") pod \"2ca278fd-dae1-47a5-8bd4-b350a4820500\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") "
Oct 30 19:47:13.266770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266736    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-token\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-token\") pod \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") "
Oct 30 19:47:13.266906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266794    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-credential-socket\") pod \"494c707f-dedc-4a62-b3fe-62aa26c6c853\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") "
Oct 30 19:47:13.266906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266829    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-token\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-token\") pod \"494c707f-dedc-4a62-b3fe-62aa26c6c853\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") "
Oct 30 19:47:13.266906 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266894    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-credential-socket\") pod \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") "
Oct 30 19:47:13.267072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266924    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-podinfo\") pod \"494c707f-dedc-4a62-b3fe-62aa26c6c853\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") "
Oct 30 19:47:13.267072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.266973    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-credential-socket\") pod \"2ca278fd-dae1-47a5-8bd4-b350a4820500\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") "
Oct 30 19:47:13.267184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267082    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-socket\") pod \"494c707f-dedc-4a62-b3fe-62aa26c6c853\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") "
Oct 30 19:47:13.267184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267124    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-token\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-token\") pod \"2ca278fd-dae1-47a5-8bd4-b350a4820500\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") "
Oct 30 19:47:13.267302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267183    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-podinfo\") pod \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") "
Oct 30 19:47:13.267302 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267264    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-kube-api-access-8zhm9\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-kube-api-access-8zhm9\") pod \"494c707f-dedc-4a62-b3fe-62aa26c6c853\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") "
Oct 30 19:47:13.267418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267301    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-certs\") pod \"494c707f-dedc-4a62-b3fe-62aa26c6c853\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") "
Oct 30 19:47:13.267418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267365    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-podinfo\") pod \"2ca278fd-dae1-47a5-8bd4-b350a4820500\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") "
Oct 30 19:47:13.267580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267419    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-socket\") pod \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") "
Oct 30 19:47:13.267580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267499    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istiod-ca-cert\") pod \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\" (UID: \"9a65bf27-a2e8-4d90-8989-fd9eeae17758\") "
Oct 30 19:47:13.267580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267531    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-kube-api-access-b627m\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-kube-api-access-b627m\") pod \"2ca278fd-dae1-47a5-8bd4-b350a4820500\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") "
Oct 30 19:47:13.267790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267580    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-envoy\") pod \"494c707f-dedc-4a62-b3fe-62aa26c6c853\" (UID: \"494c707f-dedc-4a62-b3fe-62aa26c6c853\") "
Oct 30 19:47:13.267790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.267609    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-data\") pod \"2ca278fd-dae1-47a5-8bd4-b350a4820500\" (UID: \"2ca278fd-dae1-47a5-8bd4-b350a4820500\") "
Oct 30 19:47:13.268348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.265025    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-data" pod "9a65bf27-a2e8-4d90-8989-fd9eeae17758" (UID: "9a65bf27-a2e8-4d90-8989-fd9eeae17758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.268468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.268024    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-socket" pod "2ca278fd-dae1-47a5-8bd4-b350a4820500" (UID: "2ca278fd-dae1-47a5-8bd4-b350a4820500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.268634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.268327    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca278fd-dae1-47a5-8bd4-b350a4820500-istiod-ca-cert" pod "2ca278fd-dae1-47a5-8bd4-b350a4820500" (UID: "2ca278fd-dae1-47a5-8bd4-b350a4820500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:13.268750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.268538    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-certs" pod "9a65bf27-a2e8-4d90-8989-fd9eeae17758" (UID: "9a65bf27-a2e8-4d90-8989-fd9eeae17758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.268840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.268819    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-certs" pod "2ca278fd-dae1-47a5-8bd4-b350a4820500" (UID: "2ca278fd-dae1-47a5-8bd4-b350a4820500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.269027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.268998    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494c707f-dedc-4a62-b3fe-62aa26c6c853-istiod-ca-cert" pod "494c707f-dedc-4a62-b3fe-62aa26c6c853" (UID: "494c707f-dedc-4a62-b3fe-62aa26c6c853"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:13.269295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.269272    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-data" pod "494c707f-dedc-4a62-b3fe-62aa26c6c853" (UID: "494c707f-dedc-4a62-b3fe-62aa26c6c853"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.269496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.269475    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-socket" pod "494c707f-dedc-4a62-b3fe-62aa26c6c853" (UID: "494c707f-dedc-4a62-b3fe-62aa26c6c853"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.269884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.269785    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-data" pod "2ca278fd-dae1-47a5-8bd4-b350a4820500" (UID: "2ca278fd-dae1-47a5-8bd4-b350a4820500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.269884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.269845    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-credential-socket" pod "2ca278fd-dae1-47a5-8bd4-b350a4820500" (UID: "2ca278fd-dae1-47a5-8bd4-b350a4820500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.270047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.269889    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-credential-socket" pod "494c707f-dedc-4a62-b3fe-62aa26c6c853" (UID: "494c707f-dedc-4a62-b3fe-62aa26c6c853"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.270161 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.270137    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-credential-socket" pod "9a65bf27-a2e8-4d90-8989-fd9eeae17758" (UID: "9a65bf27-a2e8-4d90-8989-fd9eeae17758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.270339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.270317    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-certs" pod "494c707f-dedc-4a62-b3fe-62aa26c6c853" (UID: "494c707f-dedc-4a62-b3fe-62aa26c6c853"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.270532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.270508    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istiod-ca-cert" pod "9a65bf27-a2e8-4d90-8989-fd9eeae17758" (UID: "9a65bf27-a2e8-4d90-8989-fd9eeae17758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:13.271986 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.271958    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-socket" pod "9a65bf27-a2e8-4d90-8989-fd9eeae17758" (UID: "9a65bf27-a2e8-4d90-8989-fd9eeae17758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.275835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.275815    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-envoy" pod "494c707f-dedc-4a62-b3fe-62aa26c6c853" (UID: "494c707f-dedc-4a62-b3fe-62aa26c6c853"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.279322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.279224    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-kube-api-access-wfnrl" pod "9a65bf27-a2e8-4d90-8989-fd9eeae17758" (UID: "9a65bf27-a2e8-4d90-8989-fd9eeae17758"). InnerVolumeSpecName "kube-api-access-wfnrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.279450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.279301    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-token" pod "2ca278fd-dae1-47a5-8bd4-b350a4820500" (UID: "2ca278fd-dae1-47a5-8bd4-b350a4820500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.280652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.280605    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:13.300373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.300346    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-kube-api-access-8zhm9" pod "494c707f-dedc-4a62-b3fe-62aa26c6c853" (UID: "494c707f-dedc-4a62-b3fe-62aa26c6c853"). InnerVolumeSpecName "kube-api-access-8zhm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.300543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.300375    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-token" pod "494c707f-dedc-4a62-b3fe-62aa26c6c853" (UID: "494c707f-dedc-4a62-b3fe-62aa26c6c853"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.300973 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.300951    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-podinfo" pod "494c707f-dedc-4a62-b3fe-62aa26c6c853" (UID: "494c707f-dedc-4a62-b3fe-62aa26c6c853"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:13.301524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.301500    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-kube-api-access-b627m" pod "2ca278fd-dae1-47a5-8bd4-b350a4820500" (UID: "2ca278fd-dae1-47a5-8bd4-b350a4820500"). InnerVolumeSpecName "kube-api-access-b627m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.302336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.302302    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-podinfo" pod "2ca278fd-dae1-47a5-8bd4-b350a4820500" (UID: "2ca278fd-dae1-47a5-8bd4-b350a4820500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:13.302832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.302797    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-envoy" pod "2ca278fd-dae1-47a5-8bd4-b350a4820500" (UID: "2ca278fd-dae1-47a5-8bd4-b350a4820500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.303060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.303030    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-token" pod "9a65bf27-a2e8-4d90-8989-fd9eeae17758" (UID: "9a65bf27-a2e8-4d90-8989-fd9eeae17758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:13.303292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.303264    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-envoy" pod "9a65bf27-a2e8-4d90-8989-fd9eeae17758" (UID: "9a65bf27-a2e8-4d90-8989-fd9eeae17758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:13.305884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.305818    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-podinfo" pod "9a65bf27-a2e8-4d90-8989-fd9eeae17758" (UID: "9a65bf27-a2e8-4d90-8989-fd9eeae17758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370118    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370151    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b627m\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-kube-api-access-b627m\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370168    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370182    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370197    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370211    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfnrl\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-kube-api-access-wfnrl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370224    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370239    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494c707f-dedc-4a62-b3fe-62aa26c6c853-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370262    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca278fd-dae1-47a5-8bd4-b350a4820500-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370276    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370291    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370307    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370323    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370340    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370353    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370370    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370384    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370393    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370402    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494c707f-dedc-4a62-b3fe-62aa26c6c853-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370411    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca278fd-dae1-47a5-8bd4-b350a4820500-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370419    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370427    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370435    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a65bf27-a2e8-4d90-8989-fd9eeae17758-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370445    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zhm9\" (UniqueName: \"kubernetes.io/projected/494c707f-dedc-4a62-b3fe-62aa26c6c853-kube-api-access-8zhm9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370453    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494c707f-dedc-4a62-b3fe-62aa26c6c853-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370462    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca278fd-dae1-47a5-8bd4-b350a4820500-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.370463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.370470    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a65bf27-a2e8-4d90-8989-fd9eeae17758-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:13.445582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.445544    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:13.663795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.662435    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:13.713708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.710339    4925 generic.go:356] "Generic (PLEG): container finished" podID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerID="26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32" exitCode=0
Oct 30 19:47:13.713708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.710440    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" event={"ID":"9a65bf27-a2e8-4d90-8989-fd9eeae17758","Type":"ContainerDied","Data":"26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32"}
Oct 30 19:47:13.713708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.710491    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84" event={"ID":"9a65bf27-a2e8-4d90-8989-fd9eeae17758","Type":"ContainerDied","Data":"4e4fc67903a3eee4288189bfd74078879112af65da5da4efd878c09a5b5eb506"}
Oct 30 19:47:13.713708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.710523    4925 scope.go:122] "RemoveContainer" containerID="26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32"
Oct 30 19:47:13.733359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.728013    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"
Oct 30 19:47:13.735506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.735479    4925 generic.go:356] "Generic (PLEG): container finished" podID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerID="e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91" exitCode=0
Oct 30 19:47:13.735760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.735597    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"
Oct 30 19:47:13.736789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.735616    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" event={"ID":"494c707f-dedc-4a62-b3fe-62aa26c6c853","Type":"ContainerDied","Data":"e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91"}
Oct 30 19:47:13.736904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.736825    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z" event={"ID":"494c707f-dedc-4a62-b3fe-62aa26c6c853","Type":"ContainerDied","Data":"9cf39aea5f4c3a7061cc4f86f55dc49018963bff5a87510fe85da52d2e75d138"}
Oct 30 19:47:13.755097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.746321    4925 generic.go:356] "Generic (PLEG): container finished" podID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerID="f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf" exitCode=0
Oct 30 19:47:13.755097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.746411    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"
Oct 30 19:47:13.755097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.746510    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" event={"ID":"2ca278fd-dae1-47a5-8bd4-b350a4820500","Type":"ContainerDied","Data":"f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf"}
Oct 30 19:47:13.755097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.746548    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr" event={"ID":"2ca278fd-dae1-47a5-8bd4-b350a4820500","Type":"ContainerDied","Data":"fa9ccdf43f16b2d3c2ce589c8f6bccb87159ed715747cdbfb77ec09fa1be0251"}
Oct 30 19:47:13.789021 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.788984    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"]
Oct 30 19:47:13.790132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.790114    4925 scope.go:122] "RemoveContainer" containerID="26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32"
Oct 30 19:47:13.791795 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.790989    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-rz27z"]
Oct 30 19:47:13.792785 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:13.792759    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32\": container with ID starting with 26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32 not found: ID does not exist" containerID="26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32"
Oct 30 19:47:13.792899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.792796    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32"} err="failed to get container status \"26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32\": rpc error: code = NotFound desc = could not find container \"26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32\": container with ID starting with 26abab7eb2f0add29180370b1db87b1184766abfa453ebd0c82b3676ea852f32 not found: ID does not exist"
Oct 30 19:47:13.792899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.792822    4925 scope.go:122] "RemoveContainer" containerID="e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91"
Oct 30 19:47:13.817440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.808267    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"]
Oct 30 19:47:13.820306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.820268    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-64d84"]
Oct 30 19:47:13.838541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.838510    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"]
Oct 30 19:47:13.844974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.843069    4925 scope.go:122] "RemoveContainer" containerID="e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91"
Oct 30 19:47:13.844974 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:13.843485    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91\": container with ID starting with e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91 not found: ID does not exist" containerID="e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91"
Oct 30 19:47:13.844974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.843519    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91"} err="failed to get container status \"e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91\": rpc error: code = NotFound desc = could not find container \"e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91\": container with ID starting with e6ae4dfbad0ba1dd4a2fafaeab8142a85ff806bb45f8fbf9508dc1d047a94f91 not found: ID does not exist"
Oct 30 19:47:13.844974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.843544    4925 scope.go:122] "RemoveContainer" containerID="f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf"
Oct 30 19:47:13.844974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.844303    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a844e-openshift-default-bcd6f4cc9-j5ljr"]
Oct 30 19:47:13.878009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.877987    4925 scope.go:122] "RemoveContainer" containerID="f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf"
Oct 30 19:47:13.878445 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:13.878419    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf\": container with ID starting with f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf not found: ID does not exist" containerID="f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf"
Oct 30 19:47:13.878545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:13.878455    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf"} err="failed to get container status \"f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf\": rpc error: code = NotFound desc = could not find container \"f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf\": container with ID starting with f2cb60cf3d84ec0af375c2dff4c2a724f57f36ce0468ff31b107b6989b33afbf not found: ID does not exist"
Oct 30 19:47:14.052603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:14.052508    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:15.065414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:15.056787    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:15.265853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:15.265822    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" path="/var/lib/kubelet/pods/2ca278fd-dae1-47a5-8bd4-b350a4820500/volumes"
Oct 30 19:47:15.266384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:15.266369    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" path="/var/lib/kubelet/pods/494c707f-dedc-4a62-b3fe-62aa26c6c853/volumes"
Oct 30 19:47:15.266845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:15.266830    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" path="/var/lib/kubelet/pods/9a65bf27-a2e8-4d90-8989-fd9eeae17758/volumes"
Oct 30 19:47:15.691466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:15.691430    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:15.859585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:15.859555    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:16.026874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:16.026734    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"]
Oct 30 19:47:16.042032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:16.039880    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad23c7-jlmst"]
Oct 30 19:47:16.049768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:16.048175    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:16.321285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:16.321190    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:16.614773 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:16.614069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:16.993768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:16.993573    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:17.036890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:17.036799    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"]
Oct 30 19:47:17.057982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:17.057953    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c117c7-wgnh4"]
Oct 30 19:47:17.269719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:17.269627    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52800a33-5c40-4d5b-aee0-2ccca0f08f1c" path="/var/lib/kubelet/pods/52800a33-5c40-4d5b-aee0-2ccca0f08f1c/volumes"
Oct 30 19:47:17.270242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:17.270222    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ac2c85-ed1e-448e-b75d-99e9b145c8bc" path="/var/lib/kubelet/pods/83ac2c85-ed1e-448e-b75d-99e9b145c8bc/volumes"
Oct 30 19:47:17.658709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:17.658622    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:18.144465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.144430    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb" podUID="9679b14b-29ef-48e7-b38b-fcb871083f90" containerName="payload-processing" containerID="cri-o://0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087" gracePeriod=25
Oct 30 19:47:18.154045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.154017    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft" podUID="c6c6447d-54be-43e8-9402-037e5dc3b1cc" containerName="payload-pre-processing" containerID="cri-o://bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7" gracePeriod=25
Oct 30 19:47:18.469784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.463511    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:18.540282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.540248    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"]
Oct 30 19:47:18.589871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.589799    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:47:18.631908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.627872    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:18.665613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.665583    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:47:18.756735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.756592    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9679b14b-29ef-48e7-b38b-fcb871083f90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9679b14b-29ef-48e7-b38b-fcb871083f90-plugins-config-volume\") pod \"9679b14b-29ef-48e7-b38b-fcb871083f90\" (UID: \"9679b14b-29ef-48e7-b38b-fcb871083f90\") "
Oct 30 19:47:18.757256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.757226    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9679b14b-29ef-48e7-b38b-fcb871083f90-kube-api-access-srhhl\" (UniqueName: \"kubernetes.io/projected/9679b14b-29ef-48e7-b38b-fcb871083f90-kube-api-access-srhhl\") pod \"9679b14b-29ef-48e7-b38b-fcb871083f90\" (UID: \"9679b14b-29ef-48e7-b38b-fcb871083f90\") "
Oct 30 19:47:18.914143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.900133    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c6447d-54be-43e8-9402-037e5dc3b1cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6447d-54be-43e8-9402-037e5dc3b1cc-plugins-config-volume\") pod \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\" (UID: \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\") "
Oct 30 19:47:18.914143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.794974    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9679b14b-29ef-48e7-b38b-fcb871083f90-plugins-config-volume" pod "9679b14b-29ef-48e7-b38b-fcb871083f90" (UID: "9679b14b-29ef-48e7-b38b-fcb871083f90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:18.934739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.934677    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c6447d-54be-43e8-9402-037e5dc3b1cc-plugins-config-volume" pod "c6c6447d-54be-43e8-9402-037e5dc3b1cc" (UID: "c6c6447d-54be-43e8-9402-037e5dc3b1cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:18.938030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.937847    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9679b14b-29ef-48e7-b38b-fcb871083f90-kube-api-access-srhhl" pod "9679b14b-29ef-48e7-b38b-fcb871083f90" (UID: "9679b14b-29ef-48e7-b38b-fcb871083f90"). InnerVolumeSpecName "kube-api-access-srhhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:18.944174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:18.944144    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:19.030736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.030630    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c6447d-54be-43e8-9402-037e5dc3b1cc-kube-api-access-5p8f9\" (UniqueName: \"kubernetes.io/projected/c6c6447d-54be-43e8-9402-037e5dc3b1cc-kube-api-access-5p8f9\") pod \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\" (UID: \"c6c6447d-54be-43e8-9402-037e5dc3b1cc\") "
Oct 30 19:47:19.064210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.064174    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhhl\" (UniqueName: \"kubernetes.io/projected/9679b14b-29ef-48e7-b38b-fcb871083f90-kube-api-access-srhhl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:19.064547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.064525    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c6447d-54be-43e8-9402-037e5dc3b1cc-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:19.064728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.064712    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9679b14b-29ef-48e7-b38b-fcb871083f90-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:19.070776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.069768    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c6447d-54be-43e8-9402-037e5dc3b1cc-kube-api-access-5p8f9" pod "c6c6447d-54be-43e8-9402-037e5dc3b1cc" (UID: "c6c6447d-54be-43e8-9402-037e5dc3b1cc"). InnerVolumeSpecName "kube-api-access-5p8f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:19.081572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.081520    4925 generic.go:356] "Generic (PLEG): container finished" podID="c6c6447d-54be-43e8-9402-037e5dc3b1cc" containerID="bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7" exitCode=0
Oct 30 19:47:19.081931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.081598    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"
Oct 30 19:47:19.082547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.081625    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft" event={"ID":"c6c6447d-54be-43e8-9402-037e5dc3b1cc","Type":"ContainerDied","Data":"bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7"}
Oct 30 19:47:19.082672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.082608    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft" event={"ID":"c6c6447d-54be-43e8-9402-037e5dc3b1cc","Type":"ContainerDied","Data":"4d5757da0c47b7cb14af695d7c840a94b9b17c7a5ae4fcb3ecedd7da507b77f7"}
Oct 30 19:47:19.082672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.082668    4925 scope.go:122] "RemoveContainer" containerID="bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7"
Oct 30 19:47:19.084888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.084868    4925 generic.go:356] "Generic (PLEG): container finished" podID="9679b14b-29ef-48e7-b38b-fcb871083f90" containerID="0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087" exitCode=0
Oct 30 19:47:19.084988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.084976    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb" event={"ID":"9679b14b-29ef-48e7-b38b-fcb871083f90","Type":"ContainerDied","Data":"0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087"}
Oct 30 19:47:19.085056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.085007    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb" event={"ID":"9679b14b-29ef-48e7-b38b-fcb871083f90","Type":"ContainerDied","Data":"2f63f24bb2defafc3663b18d7d2f3d456dd614bd3812f971171d4d35cf64bbf9"}
Oct 30 19:47:19.085056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.085055    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"
Oct 30 19:47:19.198108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.195075    4925 scope.go:122] "RemoveContainer" containerID="bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7"
Oct 30 19:47:19.198108 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:19.195563    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7\": container with ID starting with bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7 not found: ID does not exist" containerID="bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7"
Oct 30 19:47:19.198108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.195601    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7"} err="failed to get container status \"bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7\": rpc error: code = NotFound desc = could not find container \"bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7\": container with ID starting with bbd016804fda92fd9b306a2aee493c26aa395259869854d51ca513134176d0b7 not found: ID does not exist"
Oct 30 19:47:19.198108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.195627    4925 scope.go:122] "RemoveContainer" containerID="0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087"
Oct 30 19:47:19.215714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.214479    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:19.226881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.216804    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8f9\" (UniqueName: \"kubernetes.io/projected/c6c6447d-54be-43e8-9402-037e5dc3b1cc-kube-api-access-5p8f9\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:19.318738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.315227    4925 scope.go:122] "RemoveContainer" containerID="0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087"
Oct 30 19:47:19.345022 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:19.344984    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087\": container with ID starting with 0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087 not found: ID does not exist" containerID="0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087"
Oct 30 19:47:19.345300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.345273    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087"} err="failed to get container status \"0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087\": rpc error: code = NotFound desc = could not find container \"0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087\": container with ID starting with 0c55a4f2092cdf19866243dcca985e5814f495896e0679f99d1e501cc5005087 not found: ID does not exist"
Oct 30 19:47:19.356386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.348107    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"]
Oct 30 19:47:19.356708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.356671    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-7c46964458-qddjb"]
Oct 30 19:47:19.356928 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.356914    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"]
Oct 30 19:47:19.400657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.400622    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-be6b74-6b5fb9f787-wpmft"]
Oct 30 19:47:19.664196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.663288    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"]
Oct 30 19:47:19.664196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.663536    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5" podUID="e07330c4-415d-437a-bf9f-cb560ce8862c" containerName="maas-api" containerID="cri-o://ceb715b164a82c16bf0c625e26b33bbc88de94f3a942ee18648b9999401ae30b" gracePeriod=30
Oct 30 19:47:19.759727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.756528    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"]
Oct 30 19:47:19.759727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.756775    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk" podUID="b403950e-962e-4a8c-99e9-5baee57b2d55" containerName="payload-processing" containerID="cri-o://9ce64f37774b440ef826c849512d5e2c2cef86923e67a49bbfad157e21b6fa1a" gracePeriod=30
Oct 30 19:47:19.759727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.759642    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:19.774033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.767555    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"]
Oct 30 19:47:19.774033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.767780    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz" podUID="2c33c16f-f0f0-43cf-9172-ae33dc0c9125" containerName="payload-pre-processing" containerID="cri-o://24319253c6079647e5ef86b89667a127d86615275cfcfff9838b52c42da594dd" gracePeriod=30
Oct 30 19:47:19.854255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.852826    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"]
Oct 30 19:47:19.854255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.853830    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c6c6447d-54be-43e8-9402-037e5dc3b1cc" containerName="payload-pre-processing"
Oct 30 19:47:19.854255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.853851    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c6447d-54be-43e8-9402-037e5dc3b1cc" containerName="payload-pre-processing"
Oct 30 19:47:19.854255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.853880    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy"
Oct 30 19:47:19.854255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.853887    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy"
Oct 30 19:47:19.854255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.853914    4925 cpu_manager.go:414] "containerMap: removing container" podUID="29a69548-f46b-4c04-a93f-01ba17a0fa07" containerName="revoke-keys"
Oct 30 19:47:19.854255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.853922    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a69548-f46b-4c04-a93f-01ba17a0fa07" containerName="revoke-keys"
Oct 30 19:47:19.854255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.853956    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy"
Oct 30 19:47:19.854255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.853963    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854699    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9679b14b-29ef-48e7-b38b-fcb871083f90" containerName="payload-processing"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854719    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9679b14b-29ef-48e7-b38b-fcb871083f90" containerName="payload-processing"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854739    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854747    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854762    4925 cpu_manager.go:414] "containerMap: removing container" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854769    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854790    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8bd50f91-9c64-459c-a604-5cbfff4dedbf" containerName="revoke-keys"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854797    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd50f91-9c64-459c-a604-5cbfff4dedbf" containerName="revoke-keys"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854967    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9679b14b-29ef-48e7-b38b-fcb871083f90" containerName="payload-processing"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854983    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a65bf27-a2e8-4d90-8989-fd9eeae17758" containerName="istio-proxy"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.854997    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="494c707f-dedc-4a62-b3fe-62aa26c6c853" containerName="istio-proxy"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.855007    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd50f91-9c64-459c-a604-5cbfff4dedbf" containerName="revoke-keys"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.855019    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c6447d-54be-43e8-9402-037e5dc3b1cc" containerName="payload-pre-processing"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.855030    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca278fd-dae1-47a5-8bd4-b350a4820500" containerName="istio-proxy"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.855041    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a69548-f46b-4c04-a93f-01ba17a0fa07" containerName="revoke-keys"
Oct 30 19:47:19.855317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.855054    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36e9eb9-021f-4c57-ab8a-d13cc9bc5c02" containerName="istio-proxy"
Oct 30 19:47:19.860396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.860368    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:19.865874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.865835    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"]
Oct 30 19:47:19.865993 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.865861    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:19.866076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.866060    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:47:19.935444 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.935409    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:19.960323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.960280    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:19.960514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.960400    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:19.960514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:19.960496    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpx5\" (UniqueName: \"kubernetes.io/projected/d5cc272e-0844-4768-8b47-2aac36ea56d3-kube-api-access-2jpx5\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:20.065712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.062719    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:20.065712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.062830    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:20.065712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.062925    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpx5\" (UniqueName: \"kubernetes.io/projected/d5cc272e-0844-4768-8b47-2aac36ea56d3-kube-api-access-2jpx5\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:20.088711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.068477    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:20.117747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.093450    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:20.117747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.114700    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:20.147715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.124855    4925 generic.go:356] "Generic (PLEG): container finished" podID="e07330c4-415d-437a-bf9f-cb560ce8862c" containerID="ceb715b164a82c16bf0c625e26b33bbc88de94f3a942ee18648b9999401ae30b" exitCode=0
Oct 30 19:47:20.147715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.125083    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5" event={"ID":"e07330c4-415d-437a-bf9f-cb560ce8862c","Type":"ContainerDied","Data":"ceb715b164a82c16bf0c625e26b33bbc88de94f3a942ee18648b9999401ae30b"}
Oct 30 19:47:20.147715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.131620    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpx5\" (UniqueName: \"kubernetes.io/projected/d5cc272e-0844-4768-8b47-2aac36ea56d3-kube-api-access-2jpx5\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:20.147715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.132628    4925 generic.go:356] "Generic (PLEG): container finished" podID="2c33c16f-f0f0-43cf-9172-ae33dc0c9125" containerID="24319253c6079647e5ef86b89667a127d86615275cfcfff9838b52c42da594dd" exitCode=0
Oct 30 19:47:20.147715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.132744    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz" event={"ID":"2c33c16f-f0f0-43cf-9172-ae33dc0c9125","Type":"ContainerDied","Data":"24319253c6079647e5ef86b89667a127d86615275cfcfff9838b52c42da594dd"}
Oct 30 19:47:20.147715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.135276    4925 generic.go:356] "Generic (PLEG): container finished" podID="b403950e-962e-4a8c-99e9-5baee57b2d55" containerID="9ce64f37774b440ef826c849512d5e2c2cef86923e67a49bbfad157e21b6fa1a" exitCode=0
Oct 30 19:47:20.147715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.135387    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk" event={"ID":"b403950e-962e-4a8c-99e9-5baee57b2d55","Type":"ContainerDied","Data":"9ce64f37774b440ef826c849512d5e2c2cef86923e67a49bbfad157e21b6fa1a"}
Oct 30 19:47:20.250817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.249948    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:20.304298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.302291    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:20.318967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.318906    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"]
Oct 30 19:47:20.323073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.323043    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aa217-jvlcp"]
Oct 30 19:47:20.388350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.388073    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:47:20.465428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.465401    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:47:20.475758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.475735    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07330c4-415d-437a-bf9f-cb560ce8862c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07330c4-415d-437a-bf9f-cb560ce8862c-openshift-service-ca\") pod \"e07330c4-415d-437a-bf9f-cb560ce8862c\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") "
Oct 30 19:47:20.475885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.475812    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-maas-api-tls\") pod \"e07330c4-415d-437a-bf9f-cb560ce8862c\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") "
Oct 30 19:47:20.475946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.475925    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-metrics-certs\") pod \"e07330c4-415d-437a-bf9f-cb560ce8862c\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") "
Oct 30 19:47:20.476096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.476079    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07330c4-415d-437a-bf9f-cb560ce8862c-kube-api-access-6qrhr\" (UniqueName: \"kubernetes.io/projected/e07330c4-415d-437a-bf9f-cb560ce8862c-kube-api-access-6qrhr\") pod \"e07330c4-415d-437a-bf9f-cb560ce8862c\" (UID: \"e07330c4-415d-437a-bf9f-cb560ce8862c\") "
Oct 30 19:47:20.476177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.476114    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07330c4-415d-437a-bf9f-cb560ce8862c-openshift-service-ca" pod "e07330c4-415d-437a-bf9f-cb560ce8862c" (UID: "e07330c4-415d-437a-bf9f-cb560ce8862c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:20.477036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.477009    4925 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07330c4-415d-437a-bf9f-cb560ce8862c-openshift-service-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:20.478369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.478342    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07330c4-415d-437a-bf9f-cb560ce8862c-kube-api-access-6qrhr" pod "e07330c4-415d-437a-bf9f-cb560ce8862c" (UID: "e07330c4-415d-437a-bf9f-cb560ce8862c"). InnerVolumeSpecName "kube-api-access-6qrhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:20.478914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.478890    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-metrics-certs" pod "e07330c4-415d-437a-bf9f-cb560ce8862c" (UID: "e07330c4-415d-437a-bf9f-cb560ce8862c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:20.479043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.479030    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-maas-api-tls" pod "e07330c4-415d-437a-bf9f-cb560ce8862c" (UID: "e07330c4-415d-437a-bf9f-cb560ce8862c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:20.495951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.495925    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:47:20.569359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.569261    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:20.577845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.577814    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b403950e-962e-4a8c-99e9-5baee57b2d55-kube-api-access-m7kmh\" (UniqueName: \"kubernetes.io/projected/b403950e-962e-4a8c-99e9-5baee57b2d55-kube-api-access-m7kmh\") pod \"b403950e-962e-4a8c-99e9-5baee57b2d55\" (UID: \"b403950e-962e-4a8c-99e9-5baee57b2d55\") "
Oct 30 19:47:20.578009 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.577948    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-plugins-config-volume\") pod \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\" (UID: \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\") "
Oct 30 19:47:20.578123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.578105    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b403950e-962e-4a8c-99e9-5baee57b2d55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b403950e-962e-4a8c-99e9-5baee57b2d55-plugins-config-volume\") pod \"b403950e-962e-4a8c-99e9-5baee57b2d55\" (UID: \"b403950e-962e-4a8c-99e9-5baee57b2d55\") "
Oct 30 19:47:20.578196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.578175    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-kube-api-access-9vjt8\" (UniqueName: \"kubernetes.io/projected/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-kube-api-access-9vjt8\") pod \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\" (UID: \"2c33c16f-f0f0-43cf-9172-ae33dc0c9125\") "
Oct 30 19:47:20.578969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.578478    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-plugins-config-volume" pod "2c33c16f-f0f0-43cf-9172-ae33dc0c9125" (UID: "2c33c16f-f0f0-43cf-9172-ae33dc0c9125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:20.578969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.578525    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b403950e-962e-4a8c-99e9-5baee57b2d55-plugins-config-volume" pod "b403950e-962e-4a8c-99e9-5baee57b2d55" (UID: "b403950e-962e-4a8c-99e9-5baee57b2d55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:20.578969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.578873    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qrhr\" (UniqueName: \"kubernetes.io/projected/e07330c4-415d-437a-bf9f-cb560ce8862c-kube-api-access-6qrhr\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:20.578969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.578898    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b403950e-962e-4a8c-99e9-5baee57b2d55-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:20.578969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.578914    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-maas-api-tls\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:20.578969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.578930    4925 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e07330c4-415d-437a-bf9f-cb560ce8862c-metrics-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:20.578969 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.578944    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:20.580286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.580267    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b403950e-962e-4a8c-99e9-5baee57b2d55-kube-api-access-m7kmh" pod "b403950e-962e-4a8c-99e9-5baee57b2d55" (UID: "b403950e-962e-4a8c-99e9-5baee57b2d55"). InnerVolumeSpecName "kube-api-access-m7kmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:20.580738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.580720    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-kube-api-access-9vjt8" pod "2c33c16f-f0f0-43cf-9172-ae33dc0c9125" (UID: "2c33c16f-f0f0-43cf-9172-ae33dc0c9125"). InnerVolumeSpecName "kube-api-access-9vjt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:20.604224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.604181    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"]
Oct 30 19:47:20.608224 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:47:20.608197    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cc272e_0844_4768_8b47_2aac36ea56d3.slice/crio-98d4e4bee94231c29360a559b25617825f430b797439b150ea53777114f97d6c WatchSource:0}: Error finding container 98d4e4bee94231c29360a559b25617825f430b797439b150ea53777114f97d6c: Status 404 returned error can't find the container with id 98d4e4bee94231c29360a559b25617825f430b797439b150ea53777114f97d6c
Oct 30 19:47:20.680861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.680839    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7kmh\" (UniqueName: \"kubernetes.io/projected/b403950e-962e-4a8c-99e9-5baee57b2d55-kube-api-access-m7kmh\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:20.680861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.680861    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vjt8\" (UniqueName: \"kubernetes.io/projected/2c33c16f-f0f0-43cf-9172-ae33dc0c9125-kube-api-access-9vjt8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:20.789783 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:20.789736    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:21.146465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.146438    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"
Oct 30 19:47:21.146643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.146481    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5" event={"ID":"e07330c4-415d-437a-bf9f-cb560ce8862c","Type":"ContainerDied","Data":"cb6da0d91cf2ef5e7fa94c1808072d710b011fc10501092d4902a8ea31e3e8a3"}
Oct 30 19:47:21.146643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.146551    4925 scope.go:122] "RemoveContainer" containerID="ceb715b164a82c16bf0c625e26b33bbc88de94f3a942ee18648b9999401ae30b"
Oct 30 19:47:21.148868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.148832    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm" event={"ID":"d5cc272e-0844-4768-8b47-2aac36ea56d3","Type":"ContainerStarted","Data":"47f33450bc67d37408d5c8d9a3b4b4258912f4e284b92067524a59aca5a04bdf"}
Oct 30 19:47:21.149510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.148898    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm" event={"ID":"d5cc272e-0844-4768-8b47-2aac36ea56d3","Type":"ContainerStarted","Data":"98d4e4bee94231c29360a559b25617825f430b797439b150ea53777114f97d6c"}
Oct 30 19:47:21.150948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.150918    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz" event={"ID":"2c33c16f-f0f0-43cf-9172-ae33dc0c9125","Type":"ContainerDied","Data":"0de53512878f4735d55ce31d07528e1d6ca4ee797f75cc05dd8adecad49d636d"}
Oct 30 19:47:21.150948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.150923    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"
Oct 30 19:47:21.152896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.152870    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk" event={"ID":"b403950e-962e-4a8c-99e9-5baee57b2d55","Type":"ContainerDied","Data":"2f1421d2d39ffb09c1654096a2f7625873befa3c6a21254e29df46b566dfd22f"}
Oct 30 19:47:21.152896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.152890    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"
Oct 30 19:47:21.161449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.161429    4925 scope.go:122] "RemoveContainer" containerID="24319253c6079647e5ef86b89667a127d86615275cfcfff9838b52c42da594dd"
Oct 30 19:47:21.168504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.168460    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm" podStartSLOduration=2.168449075 podStartE2EDuration="2.168449075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:21.166294278 +0000 UTC m=+2476.069307316" watchObservedRunningTime="2026-10-30 19:47:21.168449075 +0000 UTC m=+2476.071462112"
Oct 30 19:47:21.185550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.185526    4925 scope.go:122] "RemoveContainer" containerID="9ce64f37774b440ef826c849512d5e2c2cef86923e67a49bbfad157e21b6fa1a"
Oct 30 19:47:21.188678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.188650    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"]
Oct 30 19:47:21.193127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.193100    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7aa217-99ffbd5c4-8bxc5"]
Oct 30 19:47:21.208887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.208862    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"]
Oct 30 19:47:21.214611 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.214589    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aa217-7bc848b57b-psqwk"]
Oct 30 19:47:21.221554 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.221532    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"]
Oct 30 19:47:21.228745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.228723    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aa217-785d964d6c-rqhfz"]
Oct 30 19:47:21.265749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.265718    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c33c16f-f0f0-43cf-9172-ae33dc0c9125" path="/var/lib/kubelet/pods/2c33c16f-f0f0-43cf-9172-ae33dc0c9125/volumes"
Oct 30 19:47:21.266207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.266192    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd50f91-9c64-459c-a604-5cbfff4dedbf" path="/var/lib/kubelet/pods/8bd50f91-9c64-459c-a604-5cbfff4dedbf/volumes"
Oct 30 19:47:21.266578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.266564    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9679b14b-29ef-48e7-b38b-fcb871083f90" path="/var/lib/kubelet/pods/9679b14b-29ef-48e7-b38b-fcb871083f90/volumes"
Oct 30 19:47:21.267281 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.267262    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b403950e-962e-4a8c-99e9-5baee57b2d55" path="/var/lib/kubelet/pods/b403950e-962e-4a8c-99e9-5baee57b2d55/volumes"
Oct 30 19:47:21.267659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.267645    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c6447d-54be-43e8-9402-037e5dc3b1cc" path="/var/lib/kubelet/pods/c6c6447d-54be-43e8-9402-037e5dc3b1cc/volumes"
Oct 30 19:47:21.268098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.268083    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07330c4-415d-437a-bf9f-cb560ce8862c" path="/var/lib/kubelet/pods/e07330c4-415d-437a-bf9f-cb560ce8862c/volumes"
Oct 30 19:47:21.600874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:21.600813    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:22.127187 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:22.127148    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:47:22.165553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:22.165518    4925 generic.go:356] "Generic (PLEG): container finished" podID="d5cc272e-0844-4768-8b47-2aac36ea56d3" containerID="47f33450bc67d37408d5c8d9a3b4b4258912f4e284b92067524a59aca5a04bdf" exitCode=0
Oct 30 19:47:22.165797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:22.165634    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm" event={"ID":"d5cc272e-0844-4768-8b47-2aac36ea56d3","Type":"ContainerDied","Data":"47f33450bc67d37408d5c8d9a3b4b4258912f4e284b92067524a59aca5a04bdf"}
Oct 30 19:47:22.540092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:22.540063    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:23.404618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.404594    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:23.541563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.541485    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8" podUID="fb02c995-3f69-4ced-b63f-f461c5c5aaf2" containerName="payload-processing" containerID="cri-o://7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7" gracePeriod=25
Oct 30 19:47:23.549632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.549602    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-xks-ca\") pod \"d5cc272e-0844-4768-8b47-2aac36ea56d3\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") "
Oct 30 19:47:23.549874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.549853    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-openshift-ca\") pod \"d5cc272e-0844-4768-8b47-2aac36ea56d3\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") "
Oct 30 19:47:23.550054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.550023    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cc272e-0844-4768-8b47-2aac36ea56d3-kube-api-access-2jpx5\" (UniqueName: \"kubernetes.io/projected/d5cc272e-0844-4768-8b47-2aac36ea56d3-kube-api-access-2jpx5\") pod \"d5cc272e-0844-4768-8b47-2aac36ea56d3\" (UID: \"d5cc272e-0844-4768-8b47-2aac36ea56d3\") "
Oct 30 19:47:23.550235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.550205    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-openshift-ca" pod "d5cc272e-0844-4768-8b47-2aac36ea56d3" (UID: "d5cc272e-0844-4768-8b47-2aac36ea56d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:23.550658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.550633    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:23.551946 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.551920    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-xks-ca" pod "d5cc272e-0844-4768-8b47-2aac36ea56d3" (UID: "d5cc272e-0844-4768-8b47-2aac36ea56d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:23.552257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.552238    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cc272e-0844-4768-8b47-2aac36ea56d3-kube-api-access-2jpx5" pod "d5cc272e-0844-4768-8b47-2aac36ea56d3" (UID: "d5cc272e-0844-4768-8b47-2aac36ea56d3"). InnerVolumeSpecName "kube-api-access-2jpx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:23.651890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.651854    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jpx5\" (UniqueName: \"kubernetes.io/projected/d5cc272e-0844-4768-8b47-2aac36ea56d3-kube-api-access-2jpx5\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:23.651890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.651890    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cc272e-0844-4768-8b47-2aac36ea56d3-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:23.694717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:23.693979    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:24.238635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:24.238605    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"
Oct 30 19:47:24.238853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:24.238640    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm" event={"ID":"d5cc272e-0844-4768-8b47-2aac36ea56d3","Type":"ContainerDied","Data":"98d4e4bee94231c29360a559b25617825f430b797439b150ea53777114f97d6c"}
Oct 30 19:47:24.238853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:24.238702    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d4e4bee94231c29360a559b25617825f430b797439b150ea53777114f97d6c"
Oct 30 19:47:24.290904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:24.290867    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:24.492096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:24.492008    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:24.610463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:24.610371    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"]
Oct 30 19:47:24.610767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:24.610717    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" containerID="cri-o://ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304" gracePeriod=30
Oct 30 19:47:24.714712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:24.714319    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:25.248367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:25.248336    4925 generic.go:356] "Generic (PLEG): container finished" podID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerID="0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d" exitCode=0
Oct 30 19:47:25.248545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:25.248413    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" event={"ID":"206b13a6-c1fc-42a0-8f34-01b580c0afbd","Type":"ContainerDied","Data":"0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d"}
Oct 30 19:47:25.248846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:25.248827    4925 scope.go:122] "RemoveContainer" containerID="0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d"
Oct 30 19:47:25.250418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:25.250388    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:25.250532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:25.250458    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:25.250532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:25.250491    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:25.665549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:25.665490    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:26.221477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.221438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:26.253445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.253410    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"]
Oct 30 19:47:26.254356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254331    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e07330c4-415d-437a-bf9f-cb560ce8862c" containerName="maas-api"
Oct 30 19:47:26.254356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254353    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07330c4-415d-437a-bf9f-cb560ce8862c" containerName="maas-api"
Oct 30 19:47:26.254576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254372    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b403950e-962e-4a8c-99e9-5baee57b2d55" containerName="payload-processing"
Oct 30 19:47:26.254576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254380    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b403950e-962e-4a8c-99e9-5baee57b2d55" containerName="payload-processing"
Oct 30 19:47:26.254576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254393    4925 cpu_manager.go:414] "containerMap: removing container" podUID="d5cc272e-0844-4768-8b47-2aac36ea56d3" containerName="revoke-keys"
Oct 30 19:47:26.254576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254402    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cc272e-0844-4768-8b47-2aac36ea56d3" containerName="revoke-keys"
Oct 30 19:47:26.254576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254473    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2c33c16f-f0f0-43cf-9172-ae33dc0c9125" containerName="payload-pre-processing"
Oct 30 19:47:26.254576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254481    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c33c16f-f0f0-43cf-9172-ae33dc0c9125" containerName="payload-pre-processing"
Oct 30 19:47:26.255025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254613    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c33c16f-f0f0-43cf-9172-ae33dc0c9125" containerName="payload-pre-processing"
Oct 30 19:47:26.255025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254628    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07330c4-415d-437a-bf9f-cb560ce8862c" containerName="maas-api"
Oct 30 19:47:26.255025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254638    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b403950e-962e-4a8c-99e9-5baee57b2d55" containerName="payload-processing"
Oct 30 19:47:26.255025 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.254649    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cc272e-0844-4768-8b47-2aac36ea56d3" containerName="revoke-keys"
Oct 30 19:47:26.258854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.258827    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:26.261953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.261922    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" event={"ID":"206b13a6-c1fc-42a0-8f34-01b580c0afbd","Type":"ContainerStarted","Data":"205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8"}
Oct 30 19:47:26.268715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.267948    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-be6b74-dockercfg-8tnmh\""
Oct 30 19:47:26.275831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.275810    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"]
Oct 30 19:47:26.347622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.347587    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-be6b74\""
Oct 30 19:47:26.405472 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.405376    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:26.429715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.427401    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfnx\" (UniqueName: \"kubernetes.io/projected/ce66c736-8ac9-4ff3-8f0f-19204795da0d-kube-api-access-6sfnx\") pod \"payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj\" (UID: \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:26.429715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.427617    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66c736-8ac9-4ff3-8f0f-19204795da0d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj\" (UID: \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:26.534760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.533106    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66c736-8ac9-4ff3-8f0f-19204795da0d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj\" (UID: \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:26.534760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.533352    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfnx\" (UniqueName: \"kubernetes.io/projected/ce66c736-8ac9-4ff3-8f0f-19204795da0d-kube-api-access-6sfnx\") pod \"payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj\" (UID: \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:26.534760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.534650    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66c736-8ac9-4ff3-8f0f-19204795da0d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj\" (UID: \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:26.565842 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.565804    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfnx\" (UniqueName: \"kubernetes.io/projected/ce66c736-8ac9-4ff3-8f0f-19204795da0d-kube-api-access-6sfnx\") pod \"payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj\" (UID: \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:26.578384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.578353    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:26.830772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.830049    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:26.922751 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:26.922725    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"]
Oct 30 19:47:26.924203 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:47:26.924177    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce66c736_8ac9_4ff3_8f0f_19204795da0d.slice/crio-ed5886e50aaf04dc683c3ae45e208312e5f1809923e271c977f0751201f922db WatchSource:0}: Error finding container ed5886e50aaf04dc683c3ae45e208312e5f1809923e271c977f0751201f922db: Status 404 returned error can't find the container with id ed5886e50aaf04dc683c3ae45e208312e5f1809923e271c977f0751201f922db
Oct 30 19:47:27.142644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:27.142612    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:47:27.143746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:27.143720    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 19:47:27.143869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:27.143769    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 19:47:27.270540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:27.270502    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj" event={"ID":"ce66c736-8ac9-4ff3-8f0f-19204795da0d","Type":"ContainerStarted","Data":"6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab"}
Oct 30 19:47:27.270848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:27.270561    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:27.270848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:27.270582    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj" event={"ID":"ce66c736-8ac9-4ff3-8f0f-19204795da0d","Type":"ContainerStarted","Data":"ed5886e50aaf04dc683c3ae45e208312e5f1809923e271c977f0751201f922db"}
Oct 30 19:47:27.290129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:27.290077    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj" podStartSLOduration=1.290059153 podStartE2EDuration="1.290059153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:27.28880285 +0000 UTC m=+2482.191815886" watchObservedRunningTime="2026-10-30 19:47:27.290059153 +0000 UTC m=+2482.193072198"
Oct 30 19:47:27.566002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:27.565517    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:28.142890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:28.142854    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 19:47:28.143301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:28.142922    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 19:47:28.280653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:28.279980    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:28.520283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:28.519833    4925 patch_prober.go:29] interesting pod/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:47:28.520283 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:28.519890    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:47:28.858799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:28.858658    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:29.042398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.042356    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:29.045211 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.045168    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"]
Oct 30 19:47:29.051231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.051203    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356d54f2-jvw25"]
Oct 30 19:47:29.151733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.150994    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 19:47:29.151733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.151056    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 19:47:29.256778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.249420    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:29.384714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.379028    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa95b6f-d809-4948-8f90-54fb7a470a77" path="/var/lib/kubelet/pods/caa95b6f-d809-4948-8f90-54fb7a470a77/volumes"
Oct 30 19:47:29.669869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.669837    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"]
Oct 30 19:47:29.674670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.674647    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.679451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.679428    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:29.679451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.679439    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:47:29.679654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.679554    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:47:29.679654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.679648    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:47:29.683573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.683550    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"]
Oct 30 19:47:29.737521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.737486    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbw5j\" (UniqueName: \"kubernetes.io/projected/c7ed1e93-37e5-47d1-88ba-842c41659d82-kube-api-access-vbw5j\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.737787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.737764    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.737918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.737853    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.841977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.840040    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.841977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.840131    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.841977 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.840258    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw5j\" (UniqueName: \"kubernetes.io/projected/c7ed1e93-37e5-47d1-88ba-842c41659d82-kube-api-access-vbw5j\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.842292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.842021    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.846502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.846470    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.854007 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.853611    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbw5j\" (UniqueName: \"kubernetes.io/projected/c7ed1e93-37e5-47d1-88ba-842c41659d82-kube-api-access-vbw5j\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:29.973919 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:29.973882    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:47:30.035522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.035487    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.042884    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-data\") pod \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") "
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.042946    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-socket\") pod \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") "
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043087    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-certs\") pod \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") "
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043147    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istiod-ca-cert\") pod \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") "
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043188    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-data" pod "e5f8dcc4-3d54-4c3a-b104-00529b0dac80" (UID: "e5f8dcc4-3d54-4c3a-b104-00529b0dac80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043231    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-credential-socket\") pod \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") "
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043275    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-token\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-token\") pod \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") "
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043332    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-kube-api-access-t4xnv\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-kube-api-access-t4xnv\") pod \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") "
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043360    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-envoy\") pod \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") "
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043388    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-socket" pod "e5f8dcc4-3d54-4c3a-b104-00529b0dac80" (UID: "e5f8dcc4-3d54-4c3a-b104-00529b0dac80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043467    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-podinfo\") pod \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\" (UID: \"e5f8dcc4-3d54-4c3a-b104-00529b0dac80\") "
Oct 30 19:47:30.043725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043592    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-credential-socket" pod "e5f8dcc4-3d54-4c3a-b104-00529b0dac80" (UID: "e5f8dcc4-3d54-4c3a-b104-00529b0dac80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:30.044374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.043833    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-certs" pod "e5f8dcc4-3d54-4c3a-b104-00529b0dac80" (UID: "e5f8dcc4-3d54-4c3a-b104-00529b0dac80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:30.044374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.044095    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.044374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.044115    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.044374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.044132    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.044374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.044146    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.044628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.044472    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istiod-ca-cert" pod "e5f8dcc4-3d54-4c3a-b104-00529b0dac80" (UID: "e5f8dcc4-3d54-4c3a-b104-00529b0dac80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:30.051207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.051181    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-token" pod "e5f8dcc4-3d54-4c3a-b104-00529b0dac80" (UID: "e5f8dcc4-3d54-4c3a-b104-00529b0dac80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:30.053403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.053375    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-podinfo" pod "e5f8dcc4-3d54-4c3a-b104-00529b0dac80" (UID: "e5f8dcc4-3d54-4c3a-b104-00529b0dac80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:30.053735 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.053681    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-envoy" pod "e5f8dcc4-3d54-4c3a-b104-00529b0dac80" (UID: "e5f8dcc4-3d54-4c3a-b104-00529b0dac80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:30.054872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.054857    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-kube-api-access-t4xnv" pod "e5f8dcc4-3d54-4c3a-b104-00529b0dac80" (UID: "e5f8dcc4-3d54-4c3a-b104-00529b0dac80"). InnerVolumeSpecName "kube-api-access-t4xnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:30.143087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.143045    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 19:47:30.143258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.143112    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 19:47:30.154670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.146474    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.154670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.146506    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4xnv\" (UniqueName: \"kubernetes.io/projected/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-kube-api-access-t4xnv\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.154670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.146522    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.154670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.146534    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.154670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.146550    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f8dcc4-3d54-4c3a-b104-00529b0dac80-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.326348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.326222    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:30.372379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.369515    4925 generic.go:356] "Generic (PLEG): container finished" podID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerID="ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304" exitCode=0
Oct 30 19:47:30.372379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.369725    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" event={"ID":"e5f8dcc4-3d54-4c3a-b104-00529b0dac80","Type":"ContainerDied","Data":"ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304"}
Oct 30 19:47:30.372379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.369782    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr" event={"ID":"e5f8dcc4-3d54-4c3a-b104-00529b0dac80","Type":"ContainerDied","Data":"3ea50a76c3d2348e366410c68d0fd97a78549608dd537ea9a96102cba105b7b5"}
Oct 30 19:47:30.372379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.369811    4925 scope.go:122] "RemoveContainer" containerID="ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304"
Oct 30 19:47:30.372379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.370001    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"
Oct 30 19:47:30.435476 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.435448    4925 scope.go:122] "RemoveContainer" containerID="ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304"
Oct 30 19:47:30.436112 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:30.436001    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304\": container with ID starting with ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304 not found: ID does not exist" containerID="ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304"
Oct 30 19:47:30.436112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.436043    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304"} err="failed to get container status \"ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304\": rpc error: code = NotFound desc = could not find container \"ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304\": container with ID starting with ab96cbf6c23e6af66e62d93f8bfa7e9d939f4a7fc6c80fc6bd2e40dd1eca0304 not found: ID does not exist"
Oct 30 19:47:30.442184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.442159    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"]
Oct 30 19:47:30.447578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.447555    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad23c7-openshift-default-7899f5cb8f-ktwlr"]
Oct 30 19:47:30.573960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.571964    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"]
Oct 30 19:47:30.675638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.675608    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:30.851392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:30.851304    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:31.069725 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.069536    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:31.161716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.161094    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body=
Oct 30 19:47:31.161716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.161153    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused"
Oct 30 19:47:31.314258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.302847    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:31.401719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.401660    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" path="/var/lib/kubelet/pods/e5f8dcc4-3d54-4c3a-b104-00529b0dac80/volumes"
Oct 30 19:47:31.407933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.407901    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"]
Oct 30 19:47:31.408183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.408165    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8e6762-d9ghg"]
Oct 30 19:47:31.456504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.456412    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8" event={"ID":"c7ed1e93-37e5-47d1-88ba-842c41659d82","Type":"ContainerStarted","Data":"a5c0a9fe7208fd26f6067c62815635174b121689cb49352a2f17e4eb41655a6c"}
Oct 30 19:47:31.456716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.456501    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8" event={"ID":"c7ed1e93-37e5-47d1-88ba-842c41659d82","Type":"ContainerStarted","Data":"8e871bfec12a7509709071ab53a024cef7524c7395c7a9aba42cacf4d29c6a5c"}
Oct 30 19:47:31.677758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.670072    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:31.899173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:31.899147    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:32.093646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:32.093603    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:32.123648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:32.123616    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:47:32.212788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:32.212712    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:47:32.238028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:32.237981    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8" podStartSLOduration=3.237966155 podStartE2EDuration="3.237966155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:31.478915332 +0000 UTC m=+2486.381928370" watchObservedRunningTime="2026-10-30 19:47:32.237966155 +0000 UTC m=+2487.140979192"
Oct 30 19:47:32.468343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:32.468268    4925 generic.go:356] "Generic (PLEG): container finished" podID="c7ed1e93-37e5-47d1-88ba-842c41659d82" containerID="a5c0a9fe7208fd26f6067c62815635174b121689cb49352a2f17e4eb41655a6c" exitCode=0
Oct 30 19:47:32.468506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:32.468350    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8" event={"ID":"c7ed1e93-37e5-47d1-88ba-842c41659d82","Type":"ContainerDied","Data":"a5c0a9fe7208fd26f6067c62815635174b121689cb49352a2f17e4eb41655a6c"}
Oct 30 19:47:32.469843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:32.469824    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:47:32.789730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:32.784278    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:33.273421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.273377    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a69548-f46b-4c04-a93f-01ba17a0fa07" path="/var/lib/kubelet/pods/29a69548-f46b-4c04-a93f-01ba17a0fa07/volumes"
Oct 30 19:47:33.406905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.406865    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8" podUID="fb02c995-3f69-4ced-b63f-f461c5c5aaf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:33.583011 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.581570    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:33.703817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.703795    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:33.784035 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.784001    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-xks-ca\") pod \"c7ed1e93-37e5-47d1-88ba-842c41659d82\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") "
Oct 30 19:47:33.784185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.784155    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-openshift-ca\") pod \"c7ed1e93-37e5-47d1-88ba-842c41659d82\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") "
Oct 30 19:47:33.784239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.784211    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ed1e93-37e5-47d1-88ba-842c41659d82-kube-api-access-vbw5j\" (UniqueName: \"kubernetes.io/projected/c7ed1e93-37e5-47d1-88ba-842c41659d82-kube-api-access-vbw5j\") pod \"c7ed1e93-37e5-47d1-88ba-842c41659d82\" (UID: \"c7ed1e93-37e5-47d1-88ba-842c41659d82\") "
Oct 30 19:47:33.784535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.784508    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-openshift-ca" pod "c7ed1e93-37e5-47d1-88ba-842c41659d82" (UID: "c7ed1e93-37e5-47d1-88ba-842c41659d82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:33.784824 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.784807    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:33.786296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.786275    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-xks-ca" pod "c7ed1e93-37e5-47d1-88ba-842c41659d82" (UID: "c7ed1e93-37e5-47d1-88ba-842c41659d82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:33.786445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.786423    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ed1e93-37e5-47d1-88ba-842c41659d82-kube-api-access-vbw5j" pod "c7ed1e93-37e5-47d1-88ba-842c41659d82" (UID: "c7ed1e93-37e5-47d1-88ba-842c41659d82"). InnerVolumeSpecName "kube-api-access-vbw5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:33.886335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.886266    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed1e93-37e5-47d1-88ba-842c41659d82-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:33.886335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:33.886296    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbw5j\" (UniqueName: \"kubernetes.io/projected/c7ed1e93-37e5-47d1-88ba-842c41659d82-kube-api-access-vbw5j\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:34.269232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:34.269186    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:34.487847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:34.487816    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"
Oct 30 19:47:34.488335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:34.487874    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8" event={"ID":"c7ed1e93-37e5-47d1-88ba-842c41659d82","Type":"ContainerDied","Data":"8e871bfec12a7509709071ab53a024cef7524c7395c7a9aba42cacf4d29c6a5c"}
Oct 30 19:47:34.488335 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:34.487931    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e871bfec12a7509709071ab53a024cef7524c7395c7a9aba42cacf4d29c6a5c"
Oct 30 19:47:35.343710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:35.341754    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:36.383796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:36.383751    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:37.001291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.001217    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"]
Oct 30 19:47:37.001823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.001788    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd" podUID="11b806c0-9eaf-4fbf-bd73-f0b59d1010a3" containerName="payload-processing" containerID="cri-o://2324d400d21687ecc0b7aafe4ce1009ce1211bb03d0c1492b546e78ce1a78778" gracePeriod=30
Oct 30 19:47:37.043337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.043304    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:37.510759 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.510720    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:37.537113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.537066    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"]
Oct 30 19:47:37.540788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.540750    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2b092-q6dgm"]
Oct 30 19:47:37.562420 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.562375    4925 generic.go:356] "Generic (PLEG): container finished" podID="11b806c0-9eaf-4fbf-bd73-f0b59d1010a3" containerID="2324d400d21687ecc0b7aafe4ce1009ce1211bb03d0c1492b546e78ce1a78778" exitCode=0
Oct 30 19:47:37.562567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.562467    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd" event={"ID":"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3","Type":"ContainerDied","Data":"2324d400d21687ecc0b7aafe4ce1009ce1211bb03d0c1492b546e78ce1a78778"}
Oct 30 19:47:37.562567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.562533    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd" event={"ID":"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3","Type":"ContainerDied","Data":"abf66262c2a95239f101330c9b5fa0ebe264e109a516ba1ff34680200c1ef0cd"}
Oct 30 19:47:37.562567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.562556    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf66262c2a95239f101330c9b5fa0ebe264e109a516ba1ff34680200c1ef0cd"
Oct 30 19:47:37.566074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.566055    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:47:37.650141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.650112    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-kube-api-access-9cn99\" (UniqueName: \"kubernetes.io/projected/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-kube-api-access-9cn99\") pod \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\" (UID: \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\") "
Oct 30 19:47:37.650346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.650220    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-plugins-config-volume\") pod \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\" (UID: \"11b806c0-9eaf-4fbf-bd73-f0b59d1010a3\") "
Oct 30 19:47:37.650617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.650595    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-plugins-config-volume" pod "11b806c0-9eaf-4fbf-bd73-f0b59d1010a3" (UID: "11b806c0-9eaf-4fbf-bd73-f0b59d1010a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:37.652575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.652550    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-kube-api-access-9cn99" pod "11b806c0-9eaf-4fbf-bd73-f0b59d1010a3" (UID: "11b806c0-9eaf-4fbf-bd73-f0b59d1010a3"). InnerVolumeSpecName "kube-api-access-9cn99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:37.697437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.697405    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:37.752547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.752510    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cn99\" (UniqueName: \"kubernetes.io/projected/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-kube-api-access-9cn99\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.752547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.752539    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:37.906528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:37.906498    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:38.287995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:38.287922    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:47:38.573200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:38.573082    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"
Oct 30 19:47:38.604830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:38.603756    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:38.630967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:38.630916    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"]
Oct 30 19:47:38.634524 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:38.634496    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2b092-54548c6cf4-2v2zd"]
Oct 30 19:47:39.263347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:39.263312    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b806c0-9eaf-4fbf-bd73-f0b59d1010a3" path="/var/lib/kubelet/pods/11b806c0-9eaf-4fbf-bd73-f0b59d1010a3/volumes"
Oct 30 19:47:39.263833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:39.263819    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cc272e-0844-4768-8b47-2aac36ea56d3" path="/var/lib/kubelet/pods/d5cc272e-0844-4768-8b47-2aac36ea56d3/volumes"
Oct 30 19:47:39.518327 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:39.518218    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:40.196315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.196283    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:40.469403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.466339    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:40.700739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.700707    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:40.909736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.896980    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"]
Oct 30 19:47:40.909736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.897204    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy" containerID="cri-o://e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27" gracePeriod=30
Oct 30 19:47:40.912432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.911776    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"]
Oct 30 19:47:40.912432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.912021    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" containerID="cri-o://3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d" gracePeriod=30
Oct 30 19:47:40.918737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.918369    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"]
Oct 30 19:47:40.918737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.918577    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy" containerID="cri-o://5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760" gracePeriod=30
Oct 30 19:47:40.925088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.922268    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"]
Oct 30 19:47:40.925088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.922490    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" containerID="cri-o://9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172" gracePeriod=30
Oct 30 19:47:40.925456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.925433    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"]
Oct 30 19:47:40.925604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:40.925585    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy" containerID="cri-o://c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd" gracePeriod=30
Oct 30 19:47:41.054711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.039636    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:41.459310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.459258    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.464732    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"]
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.465610    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c7ed1e93-37e5-47d1-88ba-842c41659d82" containerName="revoke-keys"
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.465629    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ed1e93-37e5-47d1-88ba-842c41659d82" containerName="revoke-keys"
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.465721    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy"
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.465731    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy"
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.465749    4925 cpu_manager.go:414] "containerMap: removing container" podUID="11b806c0-9eaf-4fbf-bd73-f0b59d1010a3" containerName="payload-processing"
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.465757    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b806c0-9eaf-4fbf-bd73-f0b59d1010a3" containerName="payload-processing"
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.465919    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ed1e93-37e5-47d1-88ba-842c41659d82" containerName="revoke-keys"
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.465941    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f8dcc4-3d54-4c3a-b104-00529b0dac80" containerName="istio-proxy"
Oct 30 19:47:41.469466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.465951    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b806c0-9eaf-4fbf-bd73-f0b59d1010a3" containerName="payload-processing"
Oct 30 19:47:41.482457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.482412    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"]
Oct 30 19:47:41.482643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.482577    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.486039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.486015    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:47:41.486259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.486236    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:41.486360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.486136    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:47:41.494852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.494824    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:47:41.640059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.640029    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.640059 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.640064    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.640297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.640274    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhx28\" (UniqueName: \"kubernetes.io/projected/4db24c22-b009-4de2-818e-31f13ae18f69-kube-api-access-rhx28\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.742061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.741977    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.742061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.742017    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.742279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.742259    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhx28\" (UniqueName: \"kubernetes.io/projected/4db24c22-b009-4de2-818e-31f13ae18f69-kube-api-access-rhx28\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.742787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.742726    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.744653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.744626    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.753873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.753850    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhx28\" (UniqueName: \"kubernetes.io/projected/4db24c22-b009-4de2-818e-31f13ae18f69-kube-api-access-rhx28\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:41.807034 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:41.807001    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:42.094366 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:42.094331    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"]
Oct 30 19:47:42.095940 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:47:42.095911    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db24c22_b009_4de2_818e_31f13ae18f69.slice/crio-a791371f0857202ae51d2ac0df94abe2967f85dea1ba5ea1d7939a7cb277c910 WatchSource:0}: Error finding container a791371f0857202ae51d2ac0df94abe2967f85dea1ba5ea1d7939a7cb277c910: Status 404 returned error can't find the container with id a791371f0857202ae51d2ac0df94abe2967f85dea1ba5ea1d7939a7cb277c910
Oct 30 19:47:42.253062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:42.253027    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:42.614054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:42.614014    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl" event={"ID":"4db24c22-b009-4de2-818e-31f13ae18f69","Type":"ContainerStarted","Data":"67c89b5660adaf87cdb02e0470fa7b95e260e720e700526fa9a4376f6e5d3026"}
Oct 30 19:47:42.614578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:42.614079    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl" event={"ID":"4db24c22-b009-4de2-818e-31f13ae18f69","Type":"ContainerStarted","Data":"a791371f0857202ae51d2ac0df94abe2967f85dea1ba5ea1d7939a7cb277c910"}
Oct 30 19:47:42.634024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:42.633981    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl" podStartSLOduration=1.6339674 podStartE2EDuration="1.6339674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:42.630538036 +0000 UTC m=+2497.533551073" watchObservedRunningTime="2026-10-30 19:47:42.6339674 +0000 UTC m=+2497.536980436"
Oct 30 19:47:42.971238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:42.971090    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:43.094617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.094591    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj_ce66c736-8ac9-4ff3-8f0f-19204795da0d/payload-processing/0.log"
Oct 30 19:47:43.407315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.407283    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8" podUID="fb02c995-3f69-4ced-b63f-f461c5c5aaf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:43.602670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.602632    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:43.626846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.626813    4925 generic.go:356] "Generic (PLEG): container finished" podID="4db24c22-b009-4de2-818e-31f13ae18f69" containerID="67c89b5660adaf87cdb02e0470fa7b95e260e720e700526fa9a4376f6e5d3026" exitCode=0
Oct 30 19:47:43.627248 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.626890    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl" event={"ID":"4db24c22-b009-4de2-818e-31f13ae18f69","Type":"ContainerDied","Data":"67c89b5660adaf87cdb02e0470fa7b95e260e720e700526fa9a4376f6e5d3026"}
Oct 30 19:47:43.819510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.819483    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:47:43.904897 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.904866    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-kube-api-access-mhzt2\" (UniqueName: \"kubernetes.io/projected/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-kube-api-access-mhzt2\") pod \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\" (UID: \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\") "
Oct 30 19:47:43.905065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.905029    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-plugins-config-volume\") pod \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\" (UID: \"fb02c995-3f69-4ced-b63f-f461c5c5aaf2\") "
Oct 30 19:47:43.905372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.905349    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-plugins-config-volume" pod "fb02c995-3f69-4ced-b63f-f461c5c5aaf2" (UID: "fb02c995-3f69-4ced-b63f-f461c5c5aaf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:43.905494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.905434    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:43.907119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:43.907100    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-kube-api-access-mhzt2" pod "fb02c995-3f69-4ced-b63f-f461c5c5aaf2" (UID: "fb02c995-3f69-4ced-b63f-f461c5c5aaf2"). InnerVolumeSpecName "kube-api-access-mhzt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:44.007216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.007184    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzt2\" (UniqueName: \"kubernetes.io/projected/fb02c995-3f69-4ced-b63f-f461c5c5aaf2-kube-api-access-mhzt2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:44.518394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.518210    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:44.636945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.636909    4925 generic.go:356] "Generic (PLEG): container finished" podID="fb02c995-3f69-4ced-b63f-f461c5c5aaf2" containerID="7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7" exitCode=0
Oct 30 19:47:44.637418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.636956    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"
Oct 30 19:47:44.637418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.636999    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8" event={"ID":"fb02c995-3f69-4ced-b63f-f461c5c5aaf2","Type":"ContainerDied","Data":"7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7"}
Oct 30 19:47:44.637418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.637047    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8" event={"ID":"fb02c995-3f69-4ced-b63f-f461c5c5aaf2","Type":"ContainerDied","Data":"bf4e8be510025cbec27030a0d1336e03f02b6457b05bdf6bd21fba28bdf7d0f9"}
Oct 30 19:47:44.637418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.637075    4925 scope.go:122] "RemoveContainer" containerID="7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7"
Oct 30 19:47:44.651352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.651327    4925 scope.go:122] "RemoveContainer" containerID="7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7"
Oct 30 19:47:44.651743 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:44.651713    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7\": container with ID starting with 7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7 not found: ID does not exist" containerID="7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7"
Oct 30 19:47:44.651836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.651755    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7"} err="failed to get container status \"7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7\": rpc error: code = NotFound desc = could not find container \"7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7\": container with ID starting with 7e3e6cbd0ef446b5267899c19ed8b16ab91bf436a3f5d96f6e1205ed25830fa7 not found: ID does not exist"
Oct 30 19:47:44.665542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.665506    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"]
Oct 30 19:47:44.671254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.671229    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8e6762-7745b85d8-bkft8"]
Oct 30 19:47:44.799134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.799110    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:44.816873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.816849    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db24c22-b009-4de2-818e-31f13ae18f69-kube-api-access-rhx28\" (UniqueName: \"kubernetes.io/projected/4db24c22-b009-4de2-818e-31f13ae18f69-kube-api-access-rhx28\") pod \"4db24c22-b009-4de2-818e-31f13ae18f69\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") "
Oct 30 19:47:44.817027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.816888    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-xks-ca\") pod \"4db24c22-b009-4de2-818e-31f13ae18f69\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") "
Oct 30 19:47:44.817027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.817000    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-openshift-ca\") pod \"4db24c22-b009-4de2-818e-31f13ae18f69\" (UID: \"4db24c22-b009-4de2-818e-31f13ae18f69\") "
Oct 30 19:47:44.818046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.818017    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-openshift-ca" pod "4db24c22-b009-4de2-818e-31f13ae18f69" (UID: "4db24c22-b009-4de2-818e-31f13ae18f69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:44.820617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.820583    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-xks-ca" pod "4db24c22-b009-4de2-818e-31f13ae18f69" (UID: "4db24c22-b009-4de2-818e-31f13ae18f69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:44.820923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.820903    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db24c22-b009-4de2-818e-31f13ae18f69-kube-api-access-rhx28" pod "4db24c22-b009-4de2-818e-31f13ae18f69" (UID: "4db24c22-b009-4de2-818e-31f13ae18f69"). InnerVolumeSpecName "kube-api-access-rhx28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:44.918513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.918482    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhx28\" (UniqueName: \"kubernetes.io/projected/4db24c22-b009-4de2-818e-31f13ae18f69-kube-api-access-rhx28\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:44.918513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.918511    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:44.918727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:44.918520    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db24c22-b009-4de2-818e-31f13ae18f69-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:45.094134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:45.094051    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:47:45.094134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:45.094109    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:47:45.273179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:45.273135    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb02c995-3f69-4ced-b63f-f461c5c5aaf2" path="/var/lib/kubelet/pods/fb02c995-3f69-4ced-b63f-f461c5c5aaf2/volumes"
Oct 30 19:47:45.357871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:45.357441    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:45.647924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:45.647898    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"
Oct 30 19:47:45.648332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:45.647932    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl" event={"ID":"4db24c22-b009-4de2-818e-31f13ae18f69","Type":"ContainerDied","Data":"a791371f0857202ae51d2ac0df94abe2967f85dea1ba5ea1d7939a7cb277c910"}
Oct 30 19:47:45.648332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:45.647979    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a791371f0857202ae51d2ac0df94abe2967f85dea1ba5ea1d7939a7cb277c910"
Oct 30 19:47:45.909850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:45.909756    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:46.083761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.083374    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:46.210379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.208919    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"]
Oct 30 19:47:46.210379 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.209161    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy" containerID="cri-o://2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5" gracePeriod=30
Oct 30 19:47:46.216778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.216756    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"]
Oct 30 19:47:46.217144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.217121    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy" containerID="cri-o://bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d" gracePeriod=30
Oct 30 19:47:46.349972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.349936    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:46.450478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.449143    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:47:46.545549 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.545208    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:47:46.572076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.572054    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:47:46.578259 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.578229    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:47:46.581958 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.581929    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-data\") pod \"e5842785-ef0f-4ddd-ba60-518cae3c7279\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") "
Oct 30 19:47:46.582087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.581991    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5842785-ef0f-4ddd-ba60-518cae3c7279-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5842785-ef0f-4ddd-ba60-518cae3c7279-istiod-ca-cert\") pod \"e5842785-ef0f-4ddd-ba60-518cae3c7279\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") "
Oct 30 19:47:46.582087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.582080    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-kube-api-access-jvkxt\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-kube-api-access-jvkxt\") pod \"e5842785-ef0f-4ddd-ba60-518cae3c7279\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") "
Oct 30 19:47:46.582207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.582150    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-socket\") pod \"e5842785-ef0f-4ddd-ba60-518cae3c7279\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") "
Oct 30 19:47:46.582207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.582181    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-envoy\") pod \"e5842785-ef0f-4ddd-ba60-518cae3c7279\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") "
Oct 30 19:47:46.582328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.582216    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-token\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-token\") pod \"e5842785-ef0f-4ddd-ba60-518cae3c7279\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") "
Oct 30 19:47:46.582328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.582278    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-podinfo\") pod \"e5842785-ef0f-4ddd-ba60-518cae3c7279\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") "
Oct 30 19:47:46.582328 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.582311    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-credential-socket\") pod \"e5842785-ef0f-4ddd-ba60-518cae3c7279\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") "
Oct 30 19:47:46.582539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.582522    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-certs\") pod \"e5842785-ef0f-4ddd-ba60-518cae3c7279\" (UID: \"e5842785-ef0f-4ddd-ba60-518cae3c7279\") "
Oct 30 19:47:46.583285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.583262    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-socket" pod "e5842785-ef0f-4ddd-ba60-518cae3c7279" (UID: "e5842785-ef0f-4ddd-ba60-518cae3c7279"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.583451 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.583359    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-certs" pod "e5842785-ef0f-4ddd-ba60-518cae3c7279" (UID: "e5842785-ef0f-4ddd-ba60-518cae3c7279"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.583753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.583729    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-data" pod "e5842785-ef0f-4ddd-ba60-518cae3c7279" (UID: "e5842785-ef0f-4ddd-ba60-518cae3c7279"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.583992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.583974    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5842785-ef0f-4ddd-ba60-518cae3c7279-istiod-ca-cert" pod "e5842785-ef0f-4ddd-ba60-518cae3c7279" (UID: "e5842785-ef0f-4ddd-ba60-518cae3c7279"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:46.585390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.585362    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-credential-socket" pod "e5842785-ef0f-4ddd-ba60-518cae3c7279" (UID: "e5842785-ef0f-4ddd-ba60-518cae3c7279"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.586531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.586510    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-kube-api-access-jvkxt" pod "e5842785-ef0f-4ddd-ba60-518cae3c7279" (UID: "e5842785-ef0f-4ddd-ba60-518cae3c7279"). InnerVolumeSpecName "kube-api-access-jvkxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.586662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.586641    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:47:46.588112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.588079    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-envoy" pod "e5842785-ef0f-4ddd-ba60-518cae3c7279" (UID: "e5842785-ef0f-4ddd-ba60-518cae3c7279"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.588593 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.588571    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-token" pod "e5842785-ef0f-4ddd-ba60-518cae3c7279" (UID: "e5842785-ef0f-4ddd-ba60-518cae3c7279"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.589416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.589395    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-podinfo" pod "e5842785-ef0f-4ddd-ba60-518cae3c7279" (UID: "e5842785-ef0f-4ddd-ba60-518cae3c7279"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:46.665373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.665341    4925 generic.go:356] "Generic (PLEG): container finished" podID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerID="5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760" exitCode=0
Oct 30 19:47:46.665896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.665448    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"
Oct 30 19:47:46.666036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.665470    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" event={"ID":"6aaffd65-fa65-4d9d-9e74-36dd07b6c891","Type":"ContainerDied","Data":"5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760"}
Oct 30 19:47:46.666107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.666049    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7" event={"ID":"6aaffd65-fa65-4d9d-9e74-36dd07b6c891","Type":"ContainerDied","Data":"11332e4ed297154ef6010b341f109fe4f0385d3be19b0085ae2063ba6cb5ebb8"}
Oct 30 19:47:46.666107 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.666071    4925 scope.go:122] "RemoveContainer" containerID="5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760"
Oct 30 19:47:46.668363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.668332    4925 generic.go:356] "Generic (PLEG): container finished" podID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerID="9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172" exitCode=0
Oct 30 19:47:46.668542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.668521    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" event={"ID":"24022281-55e8-4e7f-bbf2-8234ef509e19","Type":"ContainerDied","Data":"9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172"}
Oct 30 19:47:46.668599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.668573    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" event={"ID":"24022281-55e8-4e7f-bbf2-8234ef509e19","Type":"ContainerDied","Data":"b60b095ece56ec659201bce9877eb46d779c1f589ffbdbd9d0d55af160f16bcd"}
Oct 30 19:47:46.668658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.668643    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"
Oct 30 19:47:46.671116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.671092    4925 generic.go:356] "Generic (PLEG): container finished" podID="27748f04-aa17-4148-bfca-197df6076bab" containerID="c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd" exitCode=0
Oct 30 19:47:46.671230 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.671144    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"
Oct 30 19:47:46.671287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.671233    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" event={"ID":"27748f04-aa17-4148-bfca-197df6076bab","Type":"ContainerDied","Data":"c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd"}
Oct 30 19:47:46.671287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.671273    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474" event={"ID":"27748f04-aa17-4148-bfca-197df6076bab","Type":"ContainerDied","Data":"a1a6a6fea17eb5e0d1abdd7a6c4351d3fe6623c5d198bdc0b41bf496c23a5fa7"}
Oct 30 19:47:46.673057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.673032    4925 generic.go:356] "Generic (PLEG): container finished" podID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerID="e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27" exitCode=0
Oct 30 19:47:46.673156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.673142    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" event={"ID":"db7ef05f-31eb-44f2-a29e-ebc0673a13e3","Type":"ContainerDied","Data":"e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27"}
Oct 30 19:47:46.673217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.673181    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79" event={"ID":"db7ef05f-31eb-44f2-a29e-ebc0673a13e3","Type":"ContainerDied","Data":"6943b8269efb9dac6ce254f8aa1298fab6282af95dca0c8d4477d1b5f273cea3"}
Oct 30 19:47:46.673268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.673253    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"
Oct 30 19:47:46.674962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.674941    4925 generic.go:356] "Generic (PLEG): container finished" podID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerID="3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d" exitCode=0
Oct 30 19:47:46.675077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.674983    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"
Oct 30 19:47:46.675077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.675020    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" event={"ID":"e5842785-ef0f-4ddd-ba60-518cae3c7279","Type":"ContainerDied","Data":"3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d"}
Oct 30 19:47:46.675077 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.675064    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk" event={"ID":"e5842785-ef0f-4ddd-ba60-518cae3c7279","Type":"ContainerDied","Data":"16265d33a9ca6f62161fe8c88b9712e47ac9f492e986c42e2482f7aad9d313bc"}
Oct 30 19:47:46.681205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.681185    4925 scope.go:122] "RemoveContainer" containerID="5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760"
Oct 30 19:47:46.681488 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:46.681459    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760\": container with ID starting with 5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760 not found: ID does not exist" containerID="5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760"
Oct 30 19:47:46.681570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.681499    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760"} err="failed to get container status \"5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760\": rpc error: code = NotFound desc = could not find container \"5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760\": container with ID starting with 5ecd79a1deacf997cbc34d445346468aef56cddb0fc7b76bcb81138d7f5a2760 not found: ID does not exist"
Oct 30 19:47:46.681570 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.681520    4925 scope.go:122] "RemoveContainer" containerID="9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172"
Oct 30 19:47:46.683673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.683637    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-credential-socket\") pod \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") "
Oct 30 19:47:46.683804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.683769    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-certs\") pod \"27748f04-aa17-4148-bfca-197df6076bab\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") "
Oct 30 19:47:46.683873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.683819    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-istio-token\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-istio-token\") pod \"27748f04-aa17-4148-bfca-197df6076bab\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") "
Oct 30 19:47:46.683873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.683843    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-credential-socket\") pod \"27748f04-aa17-4148-bfca-197df6076bab\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") "
Oct 30 19:47:46.683999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.683872    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-podinfo\") pod \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") "
Oct 30 19:47:46.683999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.683903    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-certs\") pod \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") "
Oct 30 19:47:46.683999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.683933    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-envoy\") pod \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") "
Oct 30 19:47:46.683999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.683969    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-credential-socket\") pod \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") "
Oct 30 19:47:46.684237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684011    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-envoy\") pod \"24022281-55e8-4e7f-bbf2-8234ef509e19\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") "
Oct 30 19:47:46.684237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684044    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-certs\") pod \"24022281-55e8-4e7f-bbf2-8234ef509e19\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") "
Oct 30 19:47:46.684237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684090    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-certs" pod "27748f04-aa17-4148-bfca-197df6076bab" (UID: "27748f04-aa17-4148-bfca-197df6076bab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.684237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684125    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-socket\") pod \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") "
Oct 30 19:47:46.684237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684155    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-token\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-token\") pod \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") "
Oct 30 19:47:46.684237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684196    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-data\") pod \"24022281-55e8-4e7f-bbf2-8234ef509e19\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") "
Oct 30 19:47:46.684237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684194    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-credential-socket" pod "27748f04-aa17-4148-bfca-197df6076bab" (UID: "27748f04-aa17-4148-bfca-197df6076bab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.684237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684226    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-socket\") pod \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684277    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-credential-socket\") pod \"24022281-55e8-4e7f-bbf2-8234ef509e19\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684315    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-socket\") pod \"24022281-55e8-4e7f-bbf2-8234ef509e19\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684342    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-kube-api-access-5pzc2\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-kube-api-access-5pzc2\") pod \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684375    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-certs\") pod \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684405    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27748f04-aa17-4148-bfca-197df6076bab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27748f04-aa17-4148-bfca-197df6076bab-istiod-ca-cert\") pod \"27748f04-aa17-4148-bfca-197df6076bab\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684416    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-certs" pod "6aaffd65-fa65-4d9d-9e74-36dd07b6c891" (UID: "6aaffd65-fa65-4d9d-9e74-36dd07b6c891"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684447    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27748f04-aa17-4148-bfca-197df6076bab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27748f04-aa17-4148-bfca-197df6076bab-istio-podinfo\") pod \"27748f04-aa17-4148-bfca-197df6076bab\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684470    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-token\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-token\") pod \"24022281-55e8-4e7f-bbf2-8234ef509e19\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684470    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-socket" pod "6aaffd65-fa65-4d9d-9e74-36dd07b6c891" (UID: "6aaffd65-fa65-4d9d-9e74-36dd07b6c891"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684493    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-kube-api-access-2rj82\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-kube-api-access-2rj82\") pod \"27748f04-aa17-4148-bfca-197df6076bab\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684510    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istiod-ca-cert\") pod \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684549    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-data\") pod \"27748f04-aa17-4148-bfca-197df6076bab\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") "
Oct 30 19:47:46.684730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684655    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-certs" pod "24022281-55e8-4e7f-bbf2-8234ef509e19" (UID: "24022281-55e8-4e7f-bbf2-8234ef509e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684916    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-credential-socket" pod "db7ef05f-31eb-44f2-a29e-ebc0673a13e3" (UID: "db7ef05f-31eb-44f2-a29e-ebc0673a13e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684965    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-data\") pod \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.684997    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-data\") pod \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685035    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-podinfo\") pod \"24022281-55e8-4e7f-bbf2-8234ef509e19\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685085    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-podinfo\") pod \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685103    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istiod-ca-cert\") pod \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685120    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-credential-socket" pod "6aaffd65-fa65-4d9d-9e74-36dd07b6c891" (UID: "6aaffd65-fa65-4d9d-9e74-36dd07b6c891"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685153    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24022281-55e8-4e7f-bbf2-8234ef509e19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24022281-55e8-4e7f-bbf2-8234ef509e19-istiod-ca-cert\") pod \"24022281-55e8-4e7f-bbf2-8234ef509e19\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685181    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-socket\") pod \"27748f04-aa17-4148-bfca-197df6076bab\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685199    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-kube-api-access-2kdxd\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-kube-api-access-2kdxd\") pod \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\" (UID: \"db7ef05f-31eb-44f2-a29e-ebc0673a13e3\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685234    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-kube-api-access-v2gkw\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-kube-api-access-v2gkw\") pod \"24022281-55e8-4e7f-bbf2-8234ef509e19\" (UID: \"24022281-55e8-4e7f-bbf2-8234ef509e19\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685291    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-envoy\") pod \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") "
Oct 30 19:47:46.685437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685413    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-certs" pod "db7ef05f-31eb-44f2-a29e-ebc0673a13e3" (UID: "db7ef05f-31eb-44f2-a29e-ebc0673a13e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.686789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685586    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-token\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-token\") pod \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\" (UID: \"6aaffd65-fa65-4d9d-9e74-36dd07b6c891\") "
Oct 30 19:47:46.686789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.685725    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-envoy\") pod \"27748f04-aa17-4148-bfca-197df6076bab\" (UID: \"27748f04-aa17-4148-bfca-197df6076bab\") "
Oct 30 19:47:46.686789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.686494    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27748f04-aa17-4148-bfca-197df6076bab-istiod-ca-cert" pod "27748f04-aa17-4148-bfca-197df6076bab" (UID: "27748f04-aa17-4148-bfca-197df6076bab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:46.687120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687096    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687128    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687143    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687157    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687172    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687187    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687200    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687216    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687230    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687244    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687257    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27748f04-aa17-4148-bfca-197df6076bab-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687271    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687286    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687299    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5842785-ef0f-4ddd-ba60-518cae3c7279-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687314    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkxt\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-kube-api-access-jvkxt\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687328    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687344    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687358    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5842785-ef0f-4ddd-ba60-518cae3c7279-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.687778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.687710    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istiod-ca-cert" pod "db7ef05f-31eb-44f2-a29e-ebc0673a13e3" (UID: "db7ef05f-31eb-44f2-a29e-ebc0673a13e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:46.689765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.688087    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24022281-55e8-4e7f-bbf2-8234ef509e19-istiod-ca-cert" pod "24022281-55e8-4e7f-bbf2-8234ef509e19" (UID: "24022281-55e8-4e7f-bbf2-8234ef509e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:46.689765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.688315    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-socket" pod "27748f04-aa17-4148-bfca-197df6076bab" (UID: "27748f04-aa17-4148-bfca-197df6076bab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.691195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.690069    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27748f04-aa17-4148-bfca-197df6076bab-istio-podinfo" pod "27748f04-aa17-4148-bfca-197df6076bab" (UID: "27748f04-aa17-4148-bfca-197df6076bab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:46.691195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.690243    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-kube-api-access-5pzc2" pod "6aaffd65-fa65-4d9d-9e74-36dd07b6c891" (UID: "6aaffd65-fa65-4d9d-9e74-36dd07b6c891"). InnerVolumeSpecName "kube-api-access-5pzc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.691195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.690283    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-envoy" pod "24022281-55e8-4e7f-bbf2-8234ef509e19" (UID: "24022281-55e8-4e7f-bbf2-8234ef509e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.691195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.690529    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-data" pod "27748f04-aa17-4148-bfca-197df6076bab" (UID: "27748f04-aa17-4148-bfca-197df6076bab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.691195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.690981    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-data" pod "db7ef05f-31eb-44f2-a29e-ebc0673a13e3" (UID: "db7ef05f-31eb-44f2-a29e-ebc0673a13e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.691531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.691199    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-data" pod "6aaffd65-fa65-4d9d-9e74-36dd07b6c891" (UID: "6aaffd65-fa65-4d9d-9e74-36dd07b6c891"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.691531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.691260    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-podinfo" pod "db7ef05f-31eb-44f2-a29e-ebc0673a13e3" (UID: "db7ef05f-31eb-44f2-a29e-ebc0673a13e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:46.691653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.691570    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-podinfo" pod "6aaffd65-fa65-4d9d-9e74-36dd07b6c891" (UID: "6aaffd65-fa65-4d9d-9e74-36dd07b6c891"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:46.691908 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.691862    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-kube-api-access-2kdxd" pod "db7ef05f-31eb-44f2-a29e-ebc0673a13e3" (UID: "db7ef05f-31eb-44f2-a29e-ebc0673a13e3"). InnerVolumeSpecName "kube-api-access-2kdxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.692333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.692305    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-envoy" pod "db7ef05f-31eb-44f2-a29e-ebc0673a13e3" (UID: "db7ef05f-31eb-44f2-a29e-ebc0673a13e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.698508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.692938    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istiod-ca-cert" pod "6aaffd65-fa65-4d9d-9e74-36dd07b6c891" (UID: "6aaffd65-fa65-4d9d-9e74-36dd07b6c891"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:46.698508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.692981    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-socket" pod "db7ef05f-31eb-44f2-a29e-ebc0673a13e3" (UID: "db7ef05f-31eb-44f2-a29e-ebc0673a13e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.698508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.693123    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-data" pod "24022281-55e8-4e7f-bbf2-8234ef509e19" (UID: "24022281-55e8-4e7f-bbf2-8234ef509e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.698508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.693213    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-credential-socket" pod "24022281-55e8-4e7f-bbf2-8234ef509e19" (UID: "24022281-55e8-4e7f-bbf2-8234ef509e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.698508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.693239    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-socket" pod "24022281-55e8-4e7f-bbf2-8234ef509e19" (UID: "24022281-55e8-4e7f-bbf2-8234ef509e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.698508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.694053    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-podinfo" pod "24022281-55e8-4e7f-bbf2-8234ef509e19" (UID: "24022281-55e8-4e7f-bbf2-8234ef509e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:46.698508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.696076    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-token" pod "db7ef05f-31eb-44f2-a29e-ebc0673a13e3" (UID: "db7ef05f-31eb-44f2-a29e-ebc0673a13e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.698892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.698565    4925 scope.go:122] "RemoveContainer" containerID="9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172"
Oct 30 19:47:46.698892 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:46.698857    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172\": container with ID starting with 9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172 not found: ID does not exist" containerID="9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172"
Oct 30 19:47:46.698892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.698880    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172"} err="failed to get container status \"9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172\": rpc error: code = NotFound desc = could not find container \"9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172\": container with ID starting with 9e5d64f05e1a333dbb6edefc4e4b996d23e2adcd980b7fb12e9d1904c4deb172 not found: ID does not exist"
Oct 30 19:47:46.699036 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.698894    4925 scope.go:122] "RemoveContainer" containerID="c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd"
Oct 30 19:47:46.705199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.705177    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-kube-api-access-v2gkw" pod "24022281-55e8-4e7f-bbf2-8234ef509e19" (UID: "24022281-55e8-4e7f-bbf2-8234ef509e19"). InnerVolumeSpecName "kube-api-access-v2gkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.710724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.710568    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"]
Oct 30 19:47:46.710810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.710734    4925 scope.go:122] "RemoveContainer" containerID="c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd"
Oct 30 19:47:46.711036 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:46.711008    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd\": container with ID starting with c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd not found: ID does not exist" containerID="c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd"
Oct 30 19:47:46.711116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.711045    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd"} err="failed to get container status \"c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd\": rpc error: code = NotFound desc = could not find container \"c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd\": container with ID starting with c9d132a450d65505b7a7db7dd24d7a220249bd9089db75b9a568ad49b4dfffdd not found: ID does not exist"
Oct 30 19:47:46.711116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.711069    4925 scope.go:122] "RemoveContainer" containerID="e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27"
Oct 30 19:47:46.715929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.715024    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-envoy" pod "6aaffd65-fa65-4d9d-9e74-36dd07b6c891" (UID: "6aaffd65-fa65-4d9d-9e74-36dd07b6c891"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.715929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.715200    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-envoy" pod "27748f04-aa17-4148-bfca-197df6076bab" (UID: "27748f04-aa17-4148-bfca-197df6076bab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:46.715929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.715225    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-token" pod "24022281-55e8-4e7f-bbf2-8234ef509e19" (UID: "24022281-55e8-4e7f-bbf2-8234ef509e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.715929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.715445    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-istio-token" pod "27748f04-aa17-4148-bfca-197df6076bab" (UID: "27748f04-aa17-4148-bfca-197df6076bab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.715929 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.715843    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-kube-api-access-2rj82" pod "27748f04-aa17-4148-bfca-197df6076bab" (UID: "27748f04-aa17-4148-bfca-197df6076bab"). InnerVolumeSpecName "kube-api-access-2rj82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.716264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.715996    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-token" pod "6aaffd65-fa65-4d9d-9e74-36dd07b6c891" (UID: "6aaffd65-fa65-4d9d-9e74-36dd07b6c891"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:46.718171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.718150    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-7nfzk"]
Oct 30 19:47:46.721511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.721493    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"]
Oct 30 19:47:46.722028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722014    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy"
Oct 30 19:47:46.722075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722029    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy"
Oct 30 19:47:46.722075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722049    4925 cpu_manager.go:414] "containerMap: removing container" podUID="4db24c22-b009-4de2-818e-31f13ae18f69" containerName="revoke-keys"
Oct 30 19:47:46.722075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722053    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db24c22-b009-4de2-818e-31f13ae18f69" containerName="revoke-keys"
Oct 30 19:47:46.722075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722064    4925 cpu_manager.go:414] "containerMap: removing container" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy"
Oct 30 19:47:46.722075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722069    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722079    4925 cpu_manager.go:414] "containerMap: removing container" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722084    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722096    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722101    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722114    4925 cpu_manager.go:414] "containerMap: removing container" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722119    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722140    4925 cpu_manager.go:414] "containerMap: removing container" podUID="fb02c995-3f69-4ced-b63f-f461c5c5aaf2" containerName="payload-processing"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722144    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb02c995-3f69-4ced-b63f-f461c5c5aaf2" containerName="payload-processing"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722222    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db24c22-b009-4de2-818e-31f13ae18f69" containerName="revoke-keys"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722231    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="27748f04-aa17-4148-bfca-197df6076bab" containerName="istio-proxy"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722238    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy"
Oct 30 19:47:46.722247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722245    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" containerName="istio-proxy"
Oct 30 19:47:46.722624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722260    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" containerName="istio-proxy"
Oct 30 19:47:46.722624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722271    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb02c995-3f69-4ced-b63f-f461c5c5aaf2" containerName="payload-processing"
Oct 30 19:47:46.722624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.722277    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" containerName="istio-proxy"
Oct 30 19:47:46.724579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.724558    4925 scope.go:122] "RemoveContainer" containerID="e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27"
Oct 30 19:47:46.724901 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:46.724880    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27\": container with ID starting with e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27 not found: ID does not exist" containerID="e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27"
Oct 30 19:47:46.724982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.724907    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27"} err="failed to get container status \"e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27\": rpc error: code = NotFound desc = could not find container \"e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27\": container with ID starting with e24f2420e20bf53378c98bafb95b6f58b57f15ce1dd38859f9ef350a2bcd7e27 not found: ID does not exist"
Oct 30 19:47:46.724982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.724925    4925 scope.go:122] "RemoveContainer" containerID="3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d"
Oct 30 19:47:46.727396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.727382    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.731026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.731006    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:47:46.731142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.731037    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:47:46.733368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.733348    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"]
Oct 30 19:47:46.744730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.744090    4925 scope.go:122] "RemoveContainer" containerID="3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d"
Oct 30 19:47:46.744730 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:46.744403    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d\": container with ID starting with 3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d not found: ID does not exist" containerID="3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d"
Oct 30 19:47:46.744730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.744448    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d"} err="failed to get container status \"3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d\": rpc error: code = NotFound desc = could not find container \"3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d\": container with ID starting with 3ed701c14b19f1d975f8abc1715a3f63def5ff2e9d2dc9d84c48be87a10d3c2d not found: ID does not exist"
Oct 30 19:47:46.788463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788401    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dck57\" (UniqueName: \"kubernetes.io/projected/c7bee19a-fd98-465d-8662-6d1c738e9dbd-kube-api-access-dck57\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.788580 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788522    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.788619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788580    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.788787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788774    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788789    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788798    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788808    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788816    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788826    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788834    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788842    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788851    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24022281-55e8-4e7f-bbf2-8234ef509e19-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788859    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pzc2\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-kube-api-access-5pzc2\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788868    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27748f04-aa17-4148-bfca-197df6076bab-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788876    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788884    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rj82\" (UniqueName: \"kubernetes.io/projected/27748f04-aa17-4148-bfca-197df6076bab-kube-api-access-2rj82\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788894    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788902    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788911    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788918    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788925    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24022281-55e8-4e7f-bbf2-8234ef509e19-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788933    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788940    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788948    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24022281-55e8-4e7f-bbf2-8234ef509e19-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788957    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788964    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kdxd\" (UniqueName: \"kubernetes.io/projected/db7ef05f-31eb-44f2-a29e-ebc0673a13e3-kube-api-access-2kdxd\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788973    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2gkw\" (UniqueName: \"kubernetes.io/projected/24022281-55e8-4e7f-bbf2-8234ef509e19-kube-api-access-v2gkw\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788981    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.788982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788989    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aaffd65-fa65-4d9d-9e74-36dd07b6c891-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.789607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.788998    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27748f04-aa17-4148-bfca-197df6076bab-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.814157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.814127    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:46.814345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.814280    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:47:46.890485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.890444    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dck57\" (UniqueName: \"kubernetes.io/projected/c7bee19a-fd98-465d-8662-6d1c738e9dbd-kube-api-access-dck57\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.890671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.890547    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.890671 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.890632    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.891218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.891200    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.893151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.893133    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.902511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.902484    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dck57\" (UniqueName: \"kubernetes.io/projected/c7bee19a-fd98-465d-8662-6d1c738e9dbd-kube-api-access-dck57\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:46.990416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.990390    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"]
Oct 30 19:47:46.997512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:46.997483    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-vxkk7"]
Oct 30 19:47:47.007701 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.007652    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"]
Oct 30 19:47:47.012755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.012731    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-dr474"]
Oct 30 19:47:47.028728 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.028659    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"]
Oct 30 19:47:47.031571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.031545    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4"]
Oct 30 19:47:47.049482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.049416    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"]
Oct 30 19:47:47.052961 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.052940    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:47.061142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.061118    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-qxb79"]
Oct 30 19:47:47.137950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.137096    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:47.262599 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.262570    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" path="/var/lib/kubelet/pods/24022281-55e8-4e7f-bbf2-8234ef509e19/volumes"
Oct 30 19:47:47.263086 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.263072    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27748f04-aa17-4148-bfca-197df6076bab" path="/var/lib/kubelet/pods/27748f04-aa17-4148-bfca-197df6076bab/volumes"
Oct 30 19:47:47.263493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.263481    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aaffd65-fa65-4d9d-9e74-36dd07b6c891" path="/var/lib/kubelet/pods/6aaffd65-fa65-4d9d-9e74-36dd07b6c891/volumes"
Oct 30 19:47:47.264299 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.264283    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db7ef05f-31eb-44f2-a29e-ebc0673a13e3" path="/var/lib/kubelet/pods/db7ef05f-31eb-44f2-a29e-ebc0673a13e3/volumes"
Oct 30 19:47:47.264719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.264705    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5842785-ef0f-4ddd-ba60-518cae3c7279" path="/var/lib/kubelet/pods/e5842785-ef0f-4ddd-ba60-518cae3c7279/volumes"
Oct 30 19:47:47.280365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.280327    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:47:47.280434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.280387    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6762-openshift-default-59b8fc5d79-58bj4" podUID="24022281-55e8-4e7f-bbf2-8234ef509e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:47.379528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.379500    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"]
Oct 30 19:47:47.380943 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:47:47.380917    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bee19a_fd98_465d_8662_6d1c738e9dbd.slice/crio-4253ee1dcf1cf3267f7a0b3ed0dc74efd9db0acfd2d8206e900777a8c701a038 WatchSource:0}: Error finding container 4253ee1dcf1cf3267f7a0b3ed0dc74efd9db0acfd2d8206e900777a8c701a038: Status 404 returned error can't find the container with id 4253ee1dcf1cf3267f7a0b3ed0dc74efd9db0acfd2d8206e900777a8c701a038
Oct 30 19:47:47.691430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:47.691387    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf" event={"ID":"c7bee19a-fd98-465d-8662-6d1c738e9dbd","Type":"ContainerStarted","Data":"4253ee1dcf1cf3267f7a0b3ed0dc74efd9db0acfd2d8206e900777a8c701a038"}
Oct 30 19:47:48.107253 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:48.107172    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:48.701800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:48.701770    4925 generic.go:356] "Generic (PLEG): container finished" podID="c7bee19a-fd98-465d-8662-6d1c738e9dbd" containerID="3f62c74b25700ac07f61735b5d87fdcef5ee1efde7a177096c05fc955bcdf1d5" exitCode=0
Oct 30 19:47:48.702186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:48.701836    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf" event={"ID":"c7bee19a-fd98-465d-8662-6d1c738e9dbd","Type":"ContainerDied","Data":"3f62c74b25700ac07f61735b5d87fdcef5ee1efde7a177096c05fc955bcdf1d5"}
Oct 30 19:47:49.071521 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.071356    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:49.678641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.677041    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:49.859734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.859710    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:49.952843 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.952770    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-openshift-ca\") pod \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") "
Oct 30 19:47:49.952982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.952881    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-xks-ca\") pod \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") "
Oct 30 19:47:49.952982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.952972    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bee19a-fd98-465d-8662-6d1c738e9dbd-kube-api-access-dck57\" (UniqueName: \"kubernetes.io/projected/c7bee19a-fd98-465d-8662-6d1c738e9dbd-kube-api-access-dck57\") pod \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\" (UID: \"c7bee19a-fd98-465d-8662-6d1c738e9dbd\") "
Oct 30 19:47:49.953123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.953100    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-openshift-ca" pod "c7bee19a-fd98-465d-8662-6d1c738e9dbd" (UID: "c7bee19a-fd98-465d-8662-6d1c738e9dbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:49.953403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.953387    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:49.955176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.955154    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bee19a-fd98-465d-8662-6d1c738e9dbd-kube-api-access-dck57" pod "c7bee19a-fd98-465d-8662-6d1c738e9dbd" (UID: "c7bee19a-fd98-465d-8662-6d1c738e9dbd"). InnerVolumeSpecName "kube-api-access-dck57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:49.955260 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:49.955206    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-xks-ca" pod "c7bee19a-fd98-465d-8662-6d1c738e9dbd" (UID: "c7bee19a-fd98-465d-8662-6d1c738e9dbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:50.054740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.054677    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dck57\" (UniqueName: \"kubernetes.io/projected/c7bee19a-fd98-465d-8662-6d1c738e9dbd-kube-api-access-dck57\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:50.054740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.054737    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bee19a-fd98-465d-8662-6d1c738e9dbd-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:50.342478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.342354    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:50.515726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.514781    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:50.659904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.659734    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"]
Oct 30 19:47:50.664170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.664145    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa49c5-s2wz8"]
Oct 30 19:47:50.690493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.690455    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:50.722056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.722029    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"
Oct 30 19:47:50.722234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.722062    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf" event={"ID":"c7bee19a-fd98-465d-8662-6d1c738e9dbd","Type":"ContainerDied","Data":"4253ee1dcf1cf3267f7a0b3ed0dc74efd9db0acfd2d8206e900777a8c701a038"}
Oct 30 19:47:50.722234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:50.722118    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4253ee1dcf1cf3267f7a0b3ed0dc74efd9db0acfd2d8206e900777a8c701a038"
Oct 30 19:47:51.109310 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.109186    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:51.262479 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.262453    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ed1e93-37e5-47d1-88ba-842c41659d82" path="/var/lib/kubelet/pods/c7ed1e93-37e5-47d1-88ba-842c41659d82/volumes"
Oct 30 19:47:51.272879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.272853    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:51.518043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.518015    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:47:51.547738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.547706    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:47:51.551788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.551764    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-certs\") pod \"133d072f-aff7-4398-a15f-f065fd99ba14\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") "
Oct 30 19:47:51.551952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.551813    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-data\") pod \"133d072f-aff7-4398-a15f-f065fd99ba14\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") "
Oct 30 19:47:51.551952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.551842    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-podinfo\") pod \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") "
Oct 30 19:47:51.551952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.551883    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-socket\") pod \"133d072f-aff7-4398-a15f-f065fd99ba14\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") "
Oct 30 19:47:51.551952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.551930    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-certs\") pod \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") "
Oct 30 19:47:51.552197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.551960    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-token\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-token\") pod \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") "
Oct 30 19:47:51.552197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.551987    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133d072f-aff7-4398-a15f-f065fd99ba14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133d072f-aff7-4398-a15f-f065fd99ba14-istiod-ca-cert\") pod \"133d072f-aff7-4398-a15f-f065fd99ba14\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") "
Oct 30 19:47:51.552197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552026    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-kube-api-access-274zz\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-kube-api-access-274zz\") pod \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") "
Oct 30 19:47:51.552197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552121    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-socket\") pod \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") "
Oct 30 19:47:51.552197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552119    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-data" pod "133d072f-aff7-4398-a15f-f065fd99ba14" (UID: "133d072f-aff7-4398-a15f-f065fd99ba14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.552197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552188    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-envoy\") pod \"133d072f-aff7-4398-a15f-f065fd99ba14\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") "
Oct 30 19:47:51.552535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552199    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-socket" pod "133d072f-aff7-4398-a15f-f065fd99ba14" (UID: "133d072f-aff7-4398-a15f-f065fd99ba14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.552535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552229    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istiod-ca-cert\") pod \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") "
Oct 30 19:47:51.552535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552270    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-envoy\") pod \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") "
Oct 30 19:47:51.552535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552297    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-credential-socket\") pod \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") "
Oct 30 19:47:51.552535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552336    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-data\") pod \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\" (UID: \"7a3942c1-19e5-401e-93a1-410c6e6c9f57\") "
Oct 30 19:47:51.552535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552382    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-kube-api-access-bhfdg\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-kube-api-access-bhfdg\") pod \"133d072f-aff7-4398-a15f-f065fd99ba14\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") "
Oct 30 19:47:51.552535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552471    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-istio-token\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-istio-token\") pod \"133d072f-aff7-4398-a15f-f065fd99ba14\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") "
Oct 30 19:47:51.552535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552499    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/133d072f-aff7-4398-a15f-f065fd99ba14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133d072f-aff7-4398-a15f-f065fd99ba14-istio-podinfo\") pod \"133d072f-aff7-4398-a15f-f065fd99ba14\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") "
Oct 30 19:47:51.553003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552546    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-credential-socket\") pod \"133d072f-aff7-4398-a15f-f065fd99ba14\" (UID: \"133d072f-aff7-4398-a15f-f065fd99ba14\") "
Oct 30 19:47:51.553003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552555    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-socket" pod "7a3942c1-19e5-401e-93a1-410c6e6c9f57" (UID: "7a3942c1-19e5-401e-93a1-410c6e6c9f57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.553003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552660    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istiod-ca-cert" pod "7a3942c1-19e5-401e-93a1-410c6e6c9f57" (UID: "7a3942c1-19e5-401e-93a1-410c6e6c9f57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:51.553003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552737    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-certs" pod "133d072f-aff7-4398-a15f-f065fd99ba14" (UID: "133d072f-aff7-4398-a15f-f065fd99ba14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.553003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552784    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-credential-socket" pod "133d072f-aff7-4398-a15f-f065fd99ba14" (UID: "133d072f-aff7-4398-a15f-f065fd99ba14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.553003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.552833    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-certs" pod "7a3942c1-19e5-401e-93a1-410c6e6c9f57" (UID: "7a3942c1-19e5-401e-93a1-410c6e6c9f57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.553405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.553357    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133d072f-aff7-4398-a15f-f065fd99ba14-istiod-ca-cert" pod "133d072f-aff7-4398-a15f-f065fd99ba14" (UID: "133d072f-aff7-4398-a15f-f065fd99ba14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:51.553486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.553336    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.553552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.553533    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.553609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.553549    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.553609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.553564    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.553609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.553577    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.553609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.553591    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.553609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.553605    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.553913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.553862    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-credential-socket" pod "7a3942c1-19e5-401e-93a1-410c6e6c9f57" (UID: "7a3942c1-19e5-401e-93a1-410c6e6c9f57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.554147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.554123    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-data" pod "7a3942c1-19e5-401e-93a1-410c6e6c9f57" (UID: "7a3942c1-19e5-401e-93a1-410c6e6c9f57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.554966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.554941    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-token" pod "7a3942c1-19e5-401e-93a1-410c6e6c9f57" (UID: "7a3942c1-19e5-401e-93a1-410c6e6c9f57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.556585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.556532    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-envoy" pod "133d072f-aff7-4398-a15f-f065fd99ba14" (UID: "133d072f-aff7-4398-a15f-f065fd99ba14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.556585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.556548    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-podinfo" pod "7a3942c1-19e5-401e-93a1-410c6e6c9f57" (UID: "7a3942c1-19e5-401e-93a1-410c6e6c9f57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:51.558241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.558214    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-envoy" pod "7a3942c1-19e5-401e-93a1-410c6e6c9f57" (UID: "7a3942c1-19e5-401e-93a1-410c6e6c9f57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.558421 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.558397    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/133d072f-aff7-4398-a15f-f065fd99ba14-istio-podinfo" pod "133d072f-aff7-4398-a15f-f065fd99ba14" (UID: "133d072f-aff7-4398-a15f-f065fd99ba14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:51.558504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.558477    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-istio-token" pod "133d072f-aff7-4398-a15f-f065fd99ba14" (UID: "133d072f-aff7-4398-a15f-f065fd99ba14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.558504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.558491    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-kube-api-access-bhfdg" pod "133d072f-aff7-4398-a15f-f065fd99ba14" (UID: "133d072f-aff7-4398-a15f-f065fd99ba14"). InnerVolumeSpecName "kube-api-access-bhfdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.558615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.558413    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-kube-api-access-274zz" pod "7a3942c1-19e5-401e-93a1-410c6e6c9f57" (UID: "7a3942c1-19e5-401e-93a1-410c6e6c9f57"). InnerVolumeSpecName "kube-api-access-274zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655222    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133d072f-aff7-4398-a15f-f065fd99ba14-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655252    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655269    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655279    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655288    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhfdg\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-kube-api-access-bhfdg\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655299    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133d072f-aff7-4398-a15f-f065fd99ba14-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655307    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133d072f-aff7-4398-a15f-f065fd99ba14-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655315    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655323    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655331    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133d072f-aff7-4398-a15f-f065fd99ba14-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.658183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.655339    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-274zz\" (UniqueName: \"kubernetes.io/projected/7a3942c1-19e5-401e-93a1-410c6e6c9f57-kube-api-access-274zz\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:51.713736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.713664    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:51.730737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.730672    4925 generic.go:356] "Generic (PLEG): container finished" podID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerID="bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d" exitCode=0
Oct 30 19:47:51.730866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.730756    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"
Oct 30 19:47:51.730866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.730759    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" event={"ID":"7a3942c1-19e5-401e-93a1-410c6e6c9f57","Type":"ContainerDied","Data":"bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d"}
Oct 30 19:47:51.730866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.730837    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp" event={"ID":"7a3942c1-19e5-401e-93a1-410c6e6c9f57","Type":"ContainerDied","Data":"4ea5355f77d35e6069ec913399c482649ca6b1100d8a54b41d90b4f2bf1bcfa4"}
Oct 30 19:47:51.730866 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.730867    4925 scope.go:122] "RemoveContainer" containerID="bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d"
Oct 30 19:47:51.732391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.732366    4925 generic.go:356] "Generic (PLEG): container finished" podID="133d072f-aff7-4398-a15f-f065fd99ba14" containerID="2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5" exitCode=0
Oct 30 19:47:51.732529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.732414    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" event={"ID":"133d072f-aff7-4398-a15f-f065fd99ba14","Type":"ContainerDied","Data":"2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5"}
Oct 30 19:47:51.732529 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.732445    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq" event={"ID":"133d072f-aff7-4398-a15f-f065fd99ba14","Type":"ContainerDied","Data":"dd7aa469ed667bc456b236846ffe51456ef2792f99dd636f2006ff16853bcd2e"}
Oct 30 19:47:51.732672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.732582    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"
Oct 30 19:47:51.744015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.743999    4925 scope.go:122] "RemoveContainer" containerID="bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d"
Oct 30 19:47:51.744325 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:51.744286    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d\": container with ID starting with bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d not found: ID does not exist" containerID="bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d"
Oct 30 19:47:51.744469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.744322    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d"} err="failed to get container status \"bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d\": rpc error: code = NotFound desc = could not find container \"bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d\": container with ID starting with bf4ff52025338301936f770d0a717b4e36353589e15264abeff9c3bfcf23fa4d not found: ID does not exist"
Oct 30 19:47:51.744469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.744339    4925 scope.go:122] "RemoveContainer" containerID="2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5"
Oct 30 19:47:51.755048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.755030    4925 scope.go:122] "RemoveContainer" containerID="2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5"
Oct 30 19:47:51.755358 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:51.755329    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5\": container with ID starting with 2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5 not found: ID does not exist" containerID="2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5"
Oct 30 19:47:51.755465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.755364    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5"} err="failed to get container status \"2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5\": rpc error: code = NotFound desc = could not find container \"2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5\": container with ID starting with 2e1b500aae31d6ea8309e0e198a605a7ac9c774f908d775613e5873734e0a0f5 not found: ID does not exist"
Oct 30 19:47:51.759270 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.759249    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"]
Oct 30 19:47:51.763990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.763972    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-nwqwp"]
Oct 30 19:47:51.776731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.776706    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"]
Oct 30 19:47:51.780626 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.780603    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2b092-openshift-default-758cf88db7-hj9vq"]
Oct 30 19:47:51.885712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:51.879223    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:52.141565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:52.141438    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:52.156497 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:52.156452    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:47:52.805112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:52.805079    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:53.262507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:53.262474    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" path="/var/lib/kubelet/pods/133d072f-aff7-4398-a15f-f065fd99ba14/volumes"
Oct 30 19:47:53.262968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:53.262953    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" path="/var/lib/kubelet/pods/7a3942c1-19e5-401e-93a1-410c6e6c9f57/volumes"
Oct 30 19:47:53.446142 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:53.446071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:54.160634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:54.160598    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:54.793158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:54.793109    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:55.392400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:55.392336    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:56.175533 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:56.175500    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:56.763048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:56.762217    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:57.288950 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:57.288920    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:57.423266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:57.423233    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"]
Oct 30 19:47:57.426872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:57.426773    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1b1531-t65tl"]
Oct 30 19:47:57.527645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:57.527614    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:57.821006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:57.820973    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:57.983638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:57.983602    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:58.114342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:58.114251    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"]
Oct 30 19:47:58.119947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:58.119895    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0a27e-qzccf"]
Oct 30 19:47:58.160078 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:58.160046    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:58.373900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:58.373789    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:58.545396 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:58.545355    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:58.725713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:58.725250    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:58.877750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:58.877281    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"]
Oct 30 19:47:58.877750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:58.877497    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn" podUID="30096819-dd6b-480a-b8e4-5f8cd0302ba7" containerName="payload-processing" containerID="cri-o://5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0" gracePeriod=30
Oct 30 19:47:58.940184 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:58.939675    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:59.198718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.197043    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:59.292389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.291299    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db24c22-b009-4de2-818e-31f13ae18f69" path="/var/lib/kubelet/pods/4db24c22-b009-4de2-818e-31f13ae18f69/volumes"
Oct 30 19:47:59.292863 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.292837    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7bee19a-fd98-465d-8662-6d1c738e9dbd" path="/var/lib/kubelet/pods/c7bee19a-fd98-465d-8662-6d1c738e9dbd/volumes"
Oct 30 19:47:59.312195 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.312171    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:47:59.373062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.372998    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:59.454838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.454811    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30096819-dd6b-480a-b8e4-5f8cd0302ba7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30096819-dd6b-480a-b8e4-5f8cd0302ba7-plugins-config-volume\") pod \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\" (UID: \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\") "
Oct 30 19:47:59.455246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.454890    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30096819-dd6b-480a-b8e4-5f8cd0302ba7-kube-api-access-fbqvw\" (UniqueName: \"kubernetes.io/projected/30096819-dd6b-480a-b8e4-5f8cd0302ba7-kube-api-access-fbqvw\") pod \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\" (UID: \"30096819-dd6b-480a-b8e4-5f8cd0302ba7\") "
Oct 30 19:47:59.455290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.455261    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30096819-dd6b-480a-b8e4-5f8cd0302ba7-plugins-config-volume" pod "30096819-dd6b-480a-b8e4-5f8cd0302ba7" (UID: "30096819-dd6b-480a-b8e4-5f8cd0302ba7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:59.455464 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.455449    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30096819-dd6b-480a-b8e4-5f8cd0302ba7-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:59.457231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.457208    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30096819-dd6b-480a-b8e4-5f8cd0302ba7-kube-api-access-fbqvw" pod "30096819-dd6b-480a-b8e4-5f8cd0302ba7" (UID: "30096819-dd6b-480a-b8e4-5f8cd0302ba7"). InnerVolumeSpecName "kube-api-access-fbqvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:59.536968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.536833    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:59.561742 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.561641    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbqvw\" (UniqueName: \"kubernetes.io/projected/30096819-dd6b-480a-b8e4-5f8cd0302ba7-kube-api-access-fbqvw\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:59.729496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.729462    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:59.860100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.860006    4925 generic.go:356] "Generic (PLEG): container finished" podID="30096819-dd6b-480a-b8e4-5f8cd0302ba7" containerID="5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0" exitCode=0
Oct 30 19:47:59.860318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.860150    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn" event={"ID":"30096819-dd6b-480a-b8e4-5f8cd0302ba7","Type":"ContainerDied","Data":"5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0"}
Oct 30 19:47:59.860474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.860361    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn" event={"ID":"30096819-dd6b-480a-b8e4-5f8cd0302ba7","Type":"ContainerDied","Data":"8309652f08096c48eb2ec849ba5a97f160672d7c1e89fe15d6610321ea6612e1"}
Oct 30 19:47:59.860474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.860395    4925 scope.go:122] "RemoveContainer" containerID="5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0"
Oct 30 19:47:59.860603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.860242    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"
Oct 30 19:47:59.884649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.884626    4925 scope.go:122] "RemoveContainer" containerID="5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0"
Oct 30 19:47:59.885198 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:47:59.885177    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0\": container with ID starting with 5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0 not found: ID does not exist" containerID="5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0"
Oct 30 19:47:59.885331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.885313    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0"} err="failed to get container status \"5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0\": rpc error: code = NotFound desc = could not find container \"5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0\": container with ID starting with 5c393124f9367076c09fbc2e82a54508324a3f7f43230ec567bd797f495b64c0 not found: ID does not exist"
Oct 30 19:47:59.900162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.900132    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"]
Oct 30 19:47:59.904023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.903998    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-826518-7c7d6889c4-8gkdn"]
Oct 30 19:47:59.945888 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:47:59.945854    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:00.119585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.119485    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:00.261850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.261821    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"]
Oct 30 19:48:00.262100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.262075    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy" containerID="cri-o://a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4" gracePeriod=30
Oct 30 19:48:00.387780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.379879    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:00.711851 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.711806    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"]
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714123    4925 cpu_manager.go:414] "containerMap: removing container" podUID="30096819-dd6b-480a-b8e4-5f8cd0302ba7" containerName="payload-processing"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714149    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30096819-dd6b-480a-b8e4-5f8cd0302ba7" containerName="payload-processing"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714174    4925 cpu_manager.go:414] "containerMap: removing container" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714183    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714198    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c7bee19a-fd98-465d-8662-6d1c738e9dbd" containerName="revoke-keys"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714207    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bee19a-fd98-465d-8662-6d1c738e9dbd" containerName="revoke-keys"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714232    4925 cpu_manager.go:414] "containerMap: removing container" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714239    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714418    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="30096819-dd6b-480a-b8e4-5f8cd0302ba7" containerName="payload-processing"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714435    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="133d072f-aff7-4398-a15f-f065fd99ba14" containerName="istio-proxy"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714446    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bee19a-fd98-465d-8662-6d1c738e9dbd" containerName="revoke-keys"
Oct 30 19:48:00.716103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.714466    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3942c1-19e5-401e-93a1-410c6e6c9f57" containerName="istio-proxy"
Oct 30 19:48:00.722456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.721596    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:00.725275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.725245    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:00.726386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.726075    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"]
Oct 30 19:48:00.726386 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.726363    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:00.726810 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.726788    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:00.756799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.756771    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:00.756799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.756771    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:00.782387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.782357    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:00.782546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.782438    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn44z\" (UniqueName: \"kubernetes.io/projected/19e80a86-59c8-4c98-a056-dee68a94cc54-kube-api-access-xn44z\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:00.782546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.782515    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:00.884208 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.884174    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:00.884419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.884284    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn44z\" (UniqueName: \"kubernetes.io/projected/19e80a86-59c8-4c98-a056-dee68a94cc54-kube-api-access-xn44z\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:00.884419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.884339    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:00.884941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.884920    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:00.886923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.886904    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:00.895627 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:00.895598    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn44z\" (UniqueName: \"kubernetes.io/projected/19e80a86-59c8-4c98-a056-dee68a94cc54-kube-api-access-xn44z\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:01.045721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:01.045624    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:01.070205 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:01.070172    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:01.070367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:01.070245    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:01.262607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:01.262574    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30096819-dd6b-480a-b8e4-5f8cd0302ba7" path="/var/lib/kubelet/pods/30096819-dd6b-480a-b8e4-5f8cd0302ba7/volumes"
Oct 30 19:48:01.328764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:01.328739    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"]
Oct 30 19:48:01.332221 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:48:01.332197    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e80a86_59c8_4c98_a056_dee68a94cc54.slice/crio-e08397eb8e85051f9728b0fe264ed43eac1432125dddf174827dd252f7873c09 WatchSource:0}: Error finding container e08397eb8e85051f9728b0fe264ed43eac1432125dddf174827dd252f7873c09: Status 404 returned error can't find the container with id e08397eb8e85051f9728b0fe264ed43eac1432125dddf174827dd252f7873c09
Oct 30 19:48:01.464966 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:01.464582    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:01.877613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:01.877519    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk" event={"ID":"19e80a86-59c8-4c98-a056-dee68a94cc54","Type":"ContainerStarted","Data":"6d9c55f1ded7db86966ebb5cb6df652f44e077d5c02e99027e62d936f0ca0849"}
Oct 30 19:48:01.877613 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:01.877574    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk" event={"ID":"19e80a86-59c8-4c98-a056-dee68a94cc54","Type":"ContainerStarted","Data":"e08397eb8e85051f9728b0fe264ed43eac1432125dddf174827dd252f7873c09"}
Oct 30 19:48:01.896073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:01.896015    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk" podStartSLOduration=1.895999767 podStartE2EDuration="1.895999767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:01.893648663 +0000 UTC m=+2516.796661700" watchObservedRunningTime="2026-10-30 19:48:01.895999767 +0000 UTC m=+2516.799012803"
Oct 30 19:48:02.088158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:02.088106    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:02.738584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:02.738550    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:02.886515 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:02.886482    4925 generic.go:356] "Generic (PLEG): container finished" podID="19e80a86-59c8-4c98-a056-dee68a94cc54" containerID="6d9c55f1ded7db86966ebb5cb6df652f44e077d5c02e99027e62d936f0ca0849" exitCode=0
Oct 30 19:48:02.887008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:02.886566    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk" event={"ID":"19e80a86-59c8-4c98-a056-dee68a94cc54","Type":"ContainerDied","Data":"6d9c55f1ded7db86966ebb5cb6df652f44e077d5c02e99027e62d936f0ca0849"}
Oct 30 19:48:03.562712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:03.562124    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:04.052024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.052000    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:04.138058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.138009    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-xks-ca\") pod \"19e80a86-59c8-4c98-a056-dee68a94cc54\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") "
Oct 30 19:48:04.148614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.146923    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-openshift-ca\") pod \"19e80a86-59c8-4c98-a056-dee68a94cc54\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") "
Oct 30 19:48:04.151428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.151392    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e80a86-59c8-4c98-a056-dee68a94cc54-kube-api-access-xn44z\" (UniqueName: \"kubernetes.io/projected/19e80a86-59c8-4c98-a056-dee68a94cc54-kube-api-access-xn44z\") pod \"19e80a86-59c8-4c98-a056-dee68a94cc54\" (UID: \"19e80a86-59c8-4c98-a056-dee68a94cc54\") "
Oct 30 19:48:04.166160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.145630    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-xks-ca" pod "19e80a86-59c8-4c98-a056-dee68a94cc54" (UID: "19e80a86-59c8-4c98-a056-dee68a94cc54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:04.166160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.149141    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-openshift-ca" pod "19e80a86-59c8-4c98-a056-dee68a94cc54" (UID: "19e80a86-59c8-4c98-a056-dee68a94cc54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:04.168579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.167002    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e80a86-59c8-4c98-a056-dee68a94cc54-kube-api-access-xn44z" pod "19e80a86-59c8-4c98-a056-dee68a94cc54" (UID: "19e80a86-59c8-4c98-a056-dee68a94cc54"). InnerVolumeSpecName "kube-api-access-xn44z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:04.216337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.216307    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:04.264649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.264619    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn44z\" (UniqueName: \"kubernetes.io/projected/19e80a86-59c8-4c98-a056-dee68a94cc54-kube-api-access-xn44z\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:04.264649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.264645    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:04.264649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.264656    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e80a86-59c8-4c98-a056-dee68a94cc54-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:04.903523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.903498    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"
Oct 30 19:48:04.903766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.903530    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk" event={"ID":"19e80a86-59c8-4c98-a056-dee68a94cc54","Type":"ContainerDied","Data":"e08397eb8e85051f9728b0fe264ed43eac1432125dddf174827dd252f7873c09"}
Oct 30 19:48:04.903766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:04.903575    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e08397eb8e85051f9728b0fe264ed43eac1432125dddf174827dd252f7873c09"
Oct 30 19:48:05.065193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.065154    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:05.536073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.536005    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:48:05.624772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.624740    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:05.685103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685073    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-certs\") pod \"00256e12-693d-485f-b1ba-0f90d0065e73\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") "
Oct 30 19:48:05.685307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685135    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00256e12-693d-485f-b1ba-0f90d0065e73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00256e12-693d-485f-b1ba-0f90d0065e73-istiod-ca-cert\") pod \"00256e12-693d-485f-b1ba-0f90d0065e73\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") "
Oct 30 19:48:05.685307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685201    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-credential-socket\") pod \"00256e12-693d-485f-b1ba-0f90d0065e73\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") "
Oct 30 19:48:05.685307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685274    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00256e12-693d-485f-b1ba-0f90d0065e73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00256e12-693d-485f-b1ba-0f90d0065e73-istio-podinfo\") pod \"00256e12-693d-485f-b1ba-0f90d0065e73\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") "
Oct 30 19:48:05.685489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685311    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-envoy\") pod \"00256e12-693d-485f-b1ba-0f90d0065e73\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") "
Oct 30 19:48:05.685489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685341    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-istio-token\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-istio-token\") pod \"00256e12-693d-485f-b1ba-0f90d0065e73\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") "
Oct 30 19:48:05.685489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685371    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-socket\") pod \"00256e12-693d-485f-b1ba-0f90d0065e73\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") "
Oct 30 19:48:05.685489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685383    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-certs" pod "00256e12-693d-485f-b1ba-0f90d0065e73" (UID: "00256e12-693d-485f-b1ba-0f90d0065e73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:05.685489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685400    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-kube-api-access-mmxwh\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-kube-api-access-mmxwh\") pod \"00256e12-693d-485f-b1ba-0f90d0065e73\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") "
Oct 30 19:48:05.685818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685493    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00256e12-693d-485f-b1ba-0f90d0065e73-istiod-ca-cert" pod "00256e12-693d-485f-b1ba-0f90d0065e73" (UID: "00256e12-693d-485f-b1ba-0f90d0065e73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:05.685818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685597    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-data\") pod \"00256e12-693d-485f-b1ba-0f90d0065e73\" (UID: \"00256e12-693d-485f-b1ba-0f90d0065e73\") "
Oct 30 19:48:05.685818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685593    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-credential-socket" pod "00256e12-693d-485f-b1ba-0f90d0065e73" (UID: "00256e12-693d-485f-b1ba-0f90d0065e73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:05.685984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.685901    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-socket" pod "00256e12-693d-485f-b1ba-0f90d0065e73" (UID: "00256e12-693d-485f-b1ba-0f90d0065e73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:05.686135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.686116    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-data" pod "00256e12-693d-485f-b1ba-0f90d0065e73" (UID: "00256e12-693d-485f-b1ba-0f90d0065e73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:05.686293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.686276    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:05.686443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.686299    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:05.686443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.686312    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:05.686443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.686325    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00256e12-693d-485f-b1ba-0f90d0065e73-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:05.686443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.686340    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:05.688660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.688604    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-envoy" pod "00256e12-693d-485f-b1ba-0f90d0065e73" (UID: "00256e12-693d-485f-b1ba-0f90d0065e73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:05.688660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.688627    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00256e12-693d-485f-b1ba-0f90d0065e73-istio-podinfo" pod "00256e12-693d-485f-b1ba-0f90d0065e73" (UID: "00256e12-693d-485f-b1ba-0f90d0065e73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:05.688861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.688676    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-istio-token" pod "00256e12-693d-485f-b1ba-0f90d0065e73" (UID: "00256e12-693d-485f-b1ba-0f90d0065e73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:05.688861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.688738    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-kube-api-access-mmxwh" pod "00256e12-693d-485f-b1ba-0f90d0065e73" (UID: "00256e12-693d-485f-b1ba-0f90d0065e73"). InnerVolumeSpecName "kube-api-access-mmxwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:05.780457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.780407    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:05.787854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.787800    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00256e12-693d-485f-b1ba-0f90d0065e73-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:05.787854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.787824    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00256e12-693d-485f-b1ba-0f90d0065e73-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:05.787854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.787835    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:05.787854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.787844    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxwh\" (UniqueName: \"kubernetes.io/projected/00256e12-693d-485f-b1ba-0f90d0065e73-kube-api-access-mmxwh\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:05.898673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.898637    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"]
Oct 30 19:48:05.899269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.898902    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy" containerID="cri-o://47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0" gracePeriod=30
Oct 30 19:48:05.903959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.903799    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"]
Oct 30 19:48:05.907787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.904277    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy" containerID="cri-o://a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546" gracePeriod=30
Oct 30 19:48:05.911883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.908218    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"]
Oct 30 19:48:05.911883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.908437    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy" containerID="cri-o://e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902" gracePeriod=30
Oct 30 19:48:05.919641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.919613    4925 generic.go:356] "Generic (PLEG): container finished" podID="00256e12-693d-485f-b1ba-0f90d0065e73" containerID="a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4" exitCode=0
Oct 30 19:48:05.919818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.919652    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"
Oct 30 19:48:05.919818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.919735    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" event={"ID":"00256e12-693d-485f-b1ba-0f90d0065e73","Type":"ContainerDied","Data":"a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4"}
Oct 30 19:48:05.919818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.919800    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc" event={"ID":"00256e12-693d-485f-b1ba-0f90d0065e73","Type":"ContainerDied","Data":"c02cffb42eed7bae07e2bff7a77513327cc0bc121f8c3d760bdfcc1b9c09347a"}
Oct 30 19:48:05.920010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.919830    4925 scope.go:122] "RemoveContainer" containerID="a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4"
Oct 30 19:48:05.932654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.932630    4925 scope.go:122] "RemoveContainer" containerID="c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f"
Oct 30 19:48:05.942456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.942432    4925 scope.go:122] "RemoveContainer" containerID="a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4"
Oct 30 19:48:05.942734 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:05.942712    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4\": container with ID starting with a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4 not found: ID does not exist" containerID="a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4"
Oct 30 19:48:05.942796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.942747    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4"} err="failed to get container status \"a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4\": rpc error: code = NotFound desc = could not find container \"a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4\": container with ID starting with a6806d085ec02b56d6ebabdb7434a4bf715c9c68dd8c8de6646fc939ab2dd5a4 not found: ID does not exist"
Oct 30 19:48:05.942796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.942770    4925 scope.go:122] "RemoveContainer" containerID="c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f"
Oct 30 19:48:05.943023 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:05.943009    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f\": container with ID starting with c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f not found: ID does not exist" containerID="c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f"
Oct 30 19:48:05.943058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.943029    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f"} err="failed to get container status \"c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f\": rpc error: code = NotFound desc = could not find container \"c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f\": container with ID starting with c2e7aeb6aa896de6b69b47e27f183710c96c2502e693d2ecda27b7963421a91f not found: ID does not exist"
Oct 30 19:48:05.973723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:05.967783    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:06.040765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.037482    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"]
Oct 30 19:48:06.048711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.046215    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa49c5-openshift-default-6b56844555-6m8pc"]
Oct 30 19:48:06.052711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.050921    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:06.052711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.050999    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:06.138075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.138040    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:06.550592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.550560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:06.552254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552233    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"]
Oct 30 19:48:06.552775 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552762    4925 cpu_manager.go:414] "containerMap: removing container" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy"
Oct 30 19:48:06.552815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552777    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy"
Oct 30 19:48:06.552815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552800    4925 cpu_manager.go:414] "containerMap: removing container" podUID="19e80a86-59c8-4c98-a056-dee68a94cc54" containerName="revoke-keys"
Oct 30 19:48:06.552815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552806    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e80a86-59c8-4c98-a056-dee68a94cc54" containerName="revoke-keys"
Oct 30 19:48:06.552930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552825    4925 cpu_manager.go:414] "containerMap: removing container" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy"
Oct 30 19:48:06.552930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552831    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy"
Oct 30 19:48:06.552930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552902    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e80a86-59c8-4c98-a056-dee68a94cc54" containerName="revoke-keys"
Oct 30 19:48:06.552930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552914    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy"
Oct 30 19:48:06.552930 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.552924    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" containerName="istio-proxy"
Oct 30 19:48:06.557788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.557769    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.567265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.561241    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:06.567265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.561313    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:06.567265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.564213    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"]
Oct 30 19:48:06.607449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.607395    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:06.607672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.607528    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:06.610101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.610057    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.610387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.610369    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcdbk\" (UniqueName: \"kubernetes.io/projected/79867306-8d8e-44be-9f23-71ae7f9c5ad7-kube-api-access-vcdbk\") pod \"maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.610550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.610535    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.710341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.710311    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:06.711568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.711542    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.711724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.711703    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.711800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.711786    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcdbk\" (UniqueName: \"kubernetes.io/projected/79867306-8d8e-44be-9f23-71ae7f9c5ad7-kube-api-access-vcdbk\") pod \"maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.712202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.712181    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.714349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.714331    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.723540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.723521    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcdbk\" (UniqueName: \"kubernetes.io/projected/79867306-8d8e-44be-9f23-71ae7f9c5ad7-kube-api-access-vcdbk\") pod \"maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:06.873039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.872956    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:06.877066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:06.876908    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:07.110503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.110465    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:07.328787 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:48:07.326806    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79867306_8d8e_44be_9f23_71ae7f9c5ad7.slice/crio-1ddaabf6469e961c8c3c95b2c601cb55a714e4ab06c2b0023d19326341759128 WatchSource:0}: Error finding container 1ddaabf6469e961c8c3c95b2c601cb55a714e4ab06c2b0023d19326341759128: Status 404 returned error can't find the container with id 1ddaabf6469e961c8c3c95b2c601cb55a714e4ab06c2b0023d19326341759128
Oct 30 19:48:07.334126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.333114    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00256e12-693d-485f-b1ba-0f90d0065e73" path="/var/lib/kubelet/pods/00256e12-693d-485f-b1ba-0f90d0065e73/volumes"
Oct 30 19:48:07.334126 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.333645    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"]
Oct 30 19:48:07.338390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.337071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:07.614447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.614411    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:07.790437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.790400    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:07.818019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.817987    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"]
Oct 30 19:48:07.822261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.822236    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.832916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.832894    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.833111 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.833090    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmpz\" (UniqueName: \"kubernetes.io/projected/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-kube-api-access-dcmpz\") pod \"maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.833170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.833145    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.838132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.838109    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"]
Oct 30 19:48:07.934068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.934022    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmpz\" (UniqueName: \"kubernetes.io/projected/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-kube-api-access-dcmpz\") pod \"maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.934291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.934104    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.934291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.934184    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.934733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.934714    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.936643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.936626    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.946181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.946151    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmpz\" (UniqueName: \"kubernetes.io/projected/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-kube-api-access-dcmpz\") pod \"maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:07.953871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.953839    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq" event={"ID":"79867306-8d8e-44be-9f23-71ae7f9c5ad7","Type":"ContainerStarted","Data":"5092ec193d761c7c987cc6c3404060d765150405388f4381de1517f75f22ca64"}
Oct 30 19:48:07.953998 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.953886    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq" event={"ID":"79867306-8d8e-44be-9f23-71ae7f9c5ad7","Type":"ContainerStarted","Data":"1ddaabf6469e961c8c3c95b2c601cb55a714e4ab06c2b0023d19326341759128"}
Oct 30 19:48:07.963406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.963375    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:07.971171 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:07.971121    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq" podStartSLOduration=1.9711036 podStartE2EDuration="1.9711036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:07.970236053 +0000 UTC m=+2522.873249090" watchObservedRunningTime="2026-10-30 19:48:07.9711036 +0000 UTC m=+2522.874116638"
Oct 30 19:48:08.126516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.126474    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:08.139377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.139349    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:08.434703 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.434654    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"]
Oct 30 19:48:08.438994 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:48:08.438963    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ba9701_eed9_402f_b28c_9ebe9b6e72ae.slice/crio-eacc871960c293b365a91e10e52ea23d90e33ea5b56d551e3ac7b008f23370e4 WatchSource:0}: Error finding container eacc871960c293b365a91e10e52ea23d90e33ea5b56d551e3ac7b008f23370e4: Status 404 returned error can't find the container with id eacc871960c293b365a91e10e52ea23d90e33ea5b56d551e3ac7b008f23370e4
Oct 30 19:48:08.675112 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.675084    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:08.675289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.675137    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:08.963747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.963615    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq" event={"ID":"32ba9701-eed9-402f-b28c-9ebe9b6e72ae","Type":"ContainerStarted","Data":"e218325be94f2a91395c2a46120fe966a8a5b4785caaafe2ad110db82acc8df8"}
Oct 30 19:48:08.963747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.963700    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq" event={"ID":"32ba9701-eed9-402f-b28c-9ebe9b6e72ae","Type":"ContainerStarted","Data":"eacc871960c293b365a91e10e52ea23d90e33ea5b56d551e3ac7b008f23370e4"}
Oct 30 19:48:08.965586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.965561    4925 generic.go:356] "Generic (PLEG): container finished" podID="79867306-8d8e-44be-9f23-71ae7f9c5ad7" containerID="5092ec193d761c7c987cc6c3404060d765150405388f4381de1517f75f22ca64" exitCode=0
Oct 30 19:48:08.965736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.965643    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq" event={"ID":"79867306-8d8e-44be-9f23-71ae7f9c5ad7","Type":"ContainerDied","Data":"5092ec193d761c7c987cc6c3404060d765150405388f4381de1517f75f22ca64"}
Oct 30 19:48:08.987808 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:08.987747    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq" podStartSLOduration=1.987729335 podStartE2EDuration="1.987729335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:08.984297679 +0000 UTC m=+2523.887310710" watchObservedRunningTime="2026-10-30 19:48:08.987729335 +0000 UTC m=+2523.890742374"
Oct 30 19:48:09.045572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:09.045524    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:09.976578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:09.976548    4925 generic.go:356] "Generic (PLEG): container finished" podID="32ba9701-eed9-402f-b28c-9ebe9b6e72ae" containerID="e218325be94f2a91395c2a46120fe966a8a5b4785caaafe2ad110db82acc8df8" exitCode=0
Oct 30 19:48:09.977052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:09.976635    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq" event={"ID":"32ba9701-eed9-402f-b28c-9ebe9b6e72ae","Type":"ContainerDied","Data":"e218325be94f2a91395c2a46120fe966a8a5b4785caaafe2ad110db82acc8df8"}
Oct 30 19:48:10.047999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.047948    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:10.152258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.152235    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:10.201922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.201887    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79867306-8d8e-44be-9f23-71ae7f9c5ad7-kube-api-access-vcdbk\" (UniqueName: \"kubernetes.io/projected/79867306-8d8e-44be-9f23-71ae7f9c5ad7-kube-api-access-vcdbk\") pod \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") "
Oct 30 19:48:10.202109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.202094    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-openshift-ca\") pod \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") "
Oct 30 19:48:10.202162 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.202137    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-xks-ca\") pod \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\" (UID: \"79867306-8d8e-44be-9f23-71ae7f9c5ad7\") "
Oct 30 19:48:10.202511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.202483    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-openshift-ca" pod "79867306-8d8e-44be-9f23-71ae7f9c5ad7" (UID: "79867306-8d8e-44be-9f23-71ae7f9c5ad7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:10.202733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.202680    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:10.204290 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.204256    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79867306-8d8e-44be-9f23-71ae7f9c5ad7-kube-api-access-vcdbk" pod "79867306-8d8e-44be-9f23-71ae7f9c5ad7" (UID: "79867306-8d8e-44be-9f23-71ae7f9c5ad7"). InnerVolumeSpecName "kube-api-access-vcdbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:10.204401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.204369    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-xks-ca" pod "79867306-8d8e-44be-9f23-71ae7f9c5ad7" (UID: "79867306-8d8e-44be-9f23-71ae7f9c5ad7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:10.304028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.303955    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79867306-8d8e-44be-9f23-71ae7f9c5ad7-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:10.304028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.303980    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcdbk\" (UniqueName: \"kubernetes.io/projected/79867306-8d8e-44be-9f23-71ae7f9c5ad7-kube-api-access-vcdbk\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:10.826432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.826400    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"]
Oct 30 19:48:10.836449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.836413    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"]
Oct 30 19:48:10.962101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.962069    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:10.985355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.985333    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"
Oct 30 19:48:10.985718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.985361    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq" event={"ID":"79867306-8d8e-44be-9f23-71ae7f9c5ad7","Type":"ContainerDied","Data":"1ddaabf6469e961c8c3c95b2c601cb55a714e4ab06c2b0023d19326341759128"}
Oct 30 19:48:10.985718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:10.985408    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ddaabf6469e961c8c3c95b2c601cb55a714e4ab06c2b0023d19326341759128"
Oct 30 19:48:11.063024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.062982    4925 scope.go:122] "RemoveContainer" containerID="918f29f6a25b789ffef48fa0b3076d96b5780c1245bc22f5837e3addb9036014"
Oct 30 19:48:11.165718 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.160262    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:11.206030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.204999    4925 scope.go:122] "RemoveContainer" containerID="cd4693cd0f1a133eebd3b58f0f9353dde34cdb14844025a7786c1faa87a8a6c8"
Oct 30 19:48:11.290032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.290007    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:11.326638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.326484    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-openshift-ca\") pod \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") "
Oct 30 19:48:11.326638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.326543    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-xks-ca\") pod \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") "
Oct 30 19:48:11.326896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.326668    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-kube-api-access-dcmpz\" (UniqueName: \"kubernetes.io/projected/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-kube-api-access-dcmpz\") pod \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\" (UID: \"32ba9701-eed9-402f-b28c-9ebe9b6e72ae\") "
Oct 30 19:48:11.328089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.327942    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-openshift-ca" pod "32ba9701-eed9-402f-b28c-9ebe9b6e72ae" (UID: "32ba9701-eed9-402f-b28c-9ebe9b6e72ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:11.331145 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.331105    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-xks-ca" pod "32ba9701-eed9-402f-b28c-9ebe9b6e72ae" (UID: "32ba9701-eed9-402f-b28c-9ebe9b6e72ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:11.332061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.332024    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-kube-api-access-dcmpz" pod "32ba9701-eed9-402f-b28c-9ebe9b6e72ae" (UID: "32ba9701-eed9-402f-b28c-9ebe9b6e72ae"). InnerVolumeSpecName "kube-api-access-dcmpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.383870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.383736    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"]
Oct 30 19:48:11.389262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.389237    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:48:11.392436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.391901    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:48:11.428511 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.428488    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-envoy\") pod \"95fd71d3-52ae-490a-96a2-3676b5a508bc\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") "
Oct 30 19:48:11.428631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.428553    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-socket\") pod \"95fd71d3-52ae-490a-96a2-3676b5a508bc\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") "
Oct 30 19:48:11.428631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.428602    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-envoy\") pod \"2cdffc51-409a-4865-9be7-10e8d20e0882\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") "
Oct 30 19:48:11.428774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.428663    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-token\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-token\") pod \"95fd71d3-52ae-490a-96a2-3676b5a508bc\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") "
Oct 30 19:48:11.428905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.428884    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-socket" pod "95fd71d3-52ae-490a-96a2-3676b5a508bc" (UID: "95fd71d3-52ae-490a-96a2-3676b5a508bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.429229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429208    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-credential-socket\") pod \"2cdffc51-409a-4865-9be7-10e8d20e0882\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") "
Oct 30 19:48:11.429355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429241    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-data\") pod \"95fd71d3-52ae-490a-96a2-3676b5a508bc\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") "
Oct 30 19:48:11.429355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429338    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-kube-api-access-wz68n\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-kube-api-access-wz68n\") pod \"2cdffc51-409a-4865-9be7-10e8d20e0882\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") "
Oct 30 19:48:11.429465 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429372    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-data\") pod \"2cdffc51-409a-4865-9be7-10e8d20e0882\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") "
Oct 30 19:48:11.429556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429540    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-kube-api-access-fmf54\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-kube-api-access-fmf54\") pod \"95fd71d3-52ae-490a-96a2-3676b5a508bc\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") "
Oct 30 19:48:11.429645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429589    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-token\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-token\") pod \"2cdffc51-409a-4865-9be7-10e8d20e0882\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") "
Oct 30 19:48:11.429722 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429648    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-podinfo\") pod \"2cdffc51-409a-4865-9be7-10e8d20e0882\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") "
Oct 30 19:48:11.429788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429750    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-socket\") pod \"2cdffc51-409a-4865-9be7-10e8d20e0882\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") "
Oct 30 19:48:11.429788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429781    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95fd71d3-52ae-490a-96a2-3676b5a508bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fd71d3-52ae-490a-96a2-3676b5a508bc-istiod-ca-cert\") pod \"95fd71d3-52ae-490a-96a2-3676b5a508bc\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") "
Oct 30 19:48:11.429889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429814    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-podinfo\") pod \"95fd71d3-52ae-490a-96a2-3676b5a508bc\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") "
Oct 30 19:48:11.429889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429860    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-data" pod "95fd71d3-52ae-490a-96a2-3676b5a508bc" (UID: "95fd71d3-52ae-490a-96a2-3676b5a508bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.430006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429925    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-certs\") pod \"2cdffc51-409a-4865-9be7-10e8d20e0882\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") "
Oct 30 19:48:11.430006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.429955    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-credential-socket\") pod \"95fd71d3-52ae-490a-96a2-3676b5a508bc\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") "
Oct 30 19:48:11.430108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430007    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-credential-socket" pod "2cdffc51-409a-4865-9be7-10e8d20e0882" (UID: "2cdffc51-409a-4865-9be7-10e8d20e0882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.430108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430031    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-certs\") pod \"95fd71d3-52ae-490a-96a2-3676b5a508bc\" (UID: \"95fd71d3-52ae-490a-96a2-3676b5a508bc\") "
Oct 30 19:48:11.430108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430069    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-data" pod "2cdffc51-409a-4865-9be7-10e8d20e0882" (UID: "2cdffc51-409a-4865-9be7-10e8d20e0882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.430271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430128    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdffc51-409a-4865-9be7-10e8d20e0882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdffc51-409a-4865-9be7-10e8d20e0882-istiod-ca-cert\") pod \"2cdffc51-409a-4865-9be7-10e8d20e0882\" (UID: \"2cdffc51-409a-4865-9be7-10e8d20e0882\") "
Oct 30 19:48:11.430326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430292    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95fd71d3-52ae-490a-96a2-3676b5a508bc-istiod-ca-cert" pod "95fd71d3-52ae-490a-96a2-3676b5a508bc" (UID: "95fd71d3-52ae-490a-96a2-3676b5a508bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430890    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmpz\" (UniqueName: \"kubernetes.io/projected/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-kube-api-access-dcmpz\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430915    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430930    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fd71d3-52ae-490a-96a2-3676b5a508bc-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430944    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430959    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ba9701-eed9-402f-b28c-9ebe9b6e72ae-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430973    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.430989    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.431004    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.431222    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-certs" pod "95fd71d3-52ae-490a-96a2-3676b5a508bc" (UID: "95fd71d3-52ae-490a-96a2-3676b5a508bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.431495    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdffc51-409a-4865-9be7-10e8d20e0882-istiod-ca-cert" pod "2cdffc51-409a-4865-9be7-10e8d20e0882" (UID: "2cdffc51-409a-4865-9be7-10e8d20e0882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.431630    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-certs" pod "2cdffc51-409a-4865-9be7-10e8d20e0882" (UID: "2cdffc51-409a-4865-9be7-10e8d20e0882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.431538    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-credential-socket" pod "95fd71d3-52ae-490a-96a2-3676b5a508bc" (UID: "95fd71d3-52ae-490a-96a2-3676b5a508bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.431891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.431833    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-socket" pod "2cdffc51-409a-4865-9be7-10e8d20e0882" (UID: "2cdffc51-409a-4865-9be7-10e8d20e0882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.432604 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.432452    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-token" pod "95fd71d3-52ae-490a-96a2-3676b5a508bc" (UID: "95fd71d3-52ae-490a-96a2-3676b5a508bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.435997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.435972    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-token" pod "2cdffc51-409a-4865-9be7-10e8d20e0882" (UID: "2cdffc51-409a-4865-9be7-10e8d20e0882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.436247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.436227    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-kube-api-access-wz68n" pod "2cdffc51-409a-4865-9be7-10e8d20e0882" (UID: "2cdffc51-409a-4865-9be7-10e8d20e0882"). InnerVolumeSpecName "kube-api-access-wz68n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.436324 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.436306    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-podinfo" pod "2cdffc51-409a-4865-9be7-10e8d20e0882" (UID: "2cdffc51-409a-4865-9be7-10e8d20e0882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:11.437224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.437197    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-podinfo" pod "95fd71d3-52ae-490a-96a2-3676b5a508bc" (UID: "95fd71d3-52ae-490a-96a2-3676b5a508bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:11.438082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.438060    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-envoy" pod "95fd71d3-52ae-490a-96a2-3676b5a508bc" (UID: "95fd71d3-52ae-490a-96a2-3676b5a508bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.438807 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.438770    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-kube-api-access-fmf54" pod "95fd71d3-52ae-490a-96a2-3676b5a508bc" (UID: "95fd71d3-52ae-490a-96a2-3676b5a508bc"). InnerVolumeSpecName "kube-api-access-fmf54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.440199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.440175    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-envoy" pod "2cdffc51-409a-4865-9be7-10e8d20e0882" (UID: "2cdffc51-409a-4865-9be7-10e8d20e0882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.443030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.442863    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:48:11.523572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.523535    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:11.531552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.531525    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-socket\") pod \"94dc8df8-e90f-4697-aaa1-a332423097d2\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") "
Oct 30 19:48:11.531730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.531597    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-kube-api-access-q44v4\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-kube-api-access-q44v4\") pod \"94dc8df8-e90f-4697-aaa1-a332423097d2\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") "
Oct 30 19:48:11.531730 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.531645    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-podinfo\") pod \"94dc8df8-e90f-4697-aaa1-a332423097d2\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") "
Oct 30 19:48:11.531879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.531842    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-socket" pod "94dc8df8-e90f-4697-aaa1-a332423097d2" (UID: "94dc8df8-e90f-4697-aaa1-a332423097d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.531879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.531847    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-credential-socket\") pod \"94dc8df8-e90f-4697-aaa1-a332423097d2\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") "
Oct 30 19:48:11.532005 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.531953    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-token\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-token\") pod \"94dc8df8-e90f-4697-aaa1-a332423097d2\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") "
Oct 30 19:48:11.532075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.531989    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-certs\") pod \"94dc8df8-e90f-4697-aaa1-a332423097d2\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") "
Oct 30 19:48:11.532164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.532076    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-credential-socket" pod "94dc8df8-e90f-4697-aaa1-a332423097d2" (UID: "94dc8df8-e90f-4697-aaa1-a332423097d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.532164 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.532114    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-envoy\") pod \"94dc8df8-e90f-4697-aaa1-a332423097d2\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") "
Oct 30 19:48:11.532579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.532194    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-data\") pod \"94dc8df8-e90f-4697-aaa1-a332423097d2\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") "
Oct 30 19:48:11.532579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.532228    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-certs" pod "94dc8df8-e90f-4697-aaa1-a332423097d2" (UID: "94dc8df8-e90f-4697-aaa1-a332423097d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.532579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.532277    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94dc8df8-e90f-4697-aaa1-a332423097d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dc8df8-e90f-4697-aaa1-a332423097d2-istiod-ca-cert\") pod \"94dc8df8-e90f-4697-aaa1-a332423097d2\" (UID: \"94dc8df8-e90f-4697-aaa1-a332423097d2\") "
Oct 30 19:48:11.532579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.532500    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-data" pod "94dc8df8-e90f-4697-aaa1-a332423097d2" (UID: "94dc8df8-e90f-4697-aaa1-a332423097d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.532855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.532813    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94dc8df8-e90f-4697-aaa1-a332423097d2-istiod-ca-cert" pod "94dc8df8-e90f-4697-aaa1-a332423097d2" (UID: "94dc8df8-e90f-4697-aaa1-a332423097d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:11.533135 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533112    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533141    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533157    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533173    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94dc8df8-e90f-4697-aaa1-a332423097d2-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533187    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533202    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533216    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533232    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533246    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdffc51-409a-4865-9be7-10e8d20e0882-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533259    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533333    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533351    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533364    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533379    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz68n\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-kube-api-access-wz68n\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533395    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533409    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmf54\" (UniqueName: \"kubernetes.io/projected/95fd71d3-52ae-490a-96a2-3676b5a508bc-kube-api-access-fmf54\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533423    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.533507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.533436    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdffc51-409a-4865-9be7-10e8d20e0882-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.534509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.534489    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-token" pod "94dc8df8-e90f-4697-aaa1-a332423097d2" (UID: "94dc8df8-e90f-4697-aaa1-a332423097d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.534726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.534672    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-podinfo" pod "94dc8df8-e90f-4697-aaa1-a332423097d2" (UID: "94dc8df8-e90f-4697-aaa1-a332423097d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:11.534802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.534732    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-kube-api-access-q44v4" pod "94dc8df8-e90f-4697-aaa1-a332423097d2" (UID: "94dc8df8-e90f-4697-aaa1-a332423097d2"). InnerVolumeSpecName "kube-api-access-q44v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:11.534802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.534784    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-envoy" pod "94dc8df8-e90f-4697-aaa1-a332423097d2" (UID: "94dc8df8-e90f-4697-aaa1-a332423097d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:11.634780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.634703    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.634780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.634727    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.634780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.634737    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44v4\" (UniqueName: \"kubernetes.io/projected/94dc8df8-e90f-4697-aaa1-a332423097d2-kube-api-access-q44v4\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.634780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.634747    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94dc8df8-e90f-4697-aaa1-a332423097d2-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:11.911733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.910803    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:11.997603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.997568    4925 generic.go:356] "Generic (PLEG): container finished" podID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerID="a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546" exitCode=0
Oct 30 19:48:11.998074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.997628    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"
Oct 30 19:48:11.998074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.997659    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" event={"ID":"2cdffc51-409a-4865-9be7-10e8d20e0882","Type":"ContainerDied","Data":"a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546"}
Oct 30 19:48:11.998074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.997744    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2" event={"ID":"2cdffc51-409a-4865-9be7-10e8d20e0882","Type":"ContainerDied","Data":"078c5e1ce7b21fcab8e78bd03b790c87e7eab50e834ce7c27c6f11b54408b4c8"}
Oct 30 19:48:11.998074 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.997775    4925 scope.go:122] "RemoveContainer" containerID="a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546"
Oct 30 19:48:11.999748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.999721    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq" event={"ID":"32ba9701-eed9-402f-b28c-9ebe9b6e72ae","Type":"ContainerDied","Data":"eacc871960c293b365a91e10e52ea23d90e33ea5b56d551e3ac7b008f23370e4"}
Oct 30 19:48:11.999865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.999751    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"
Oct 30 19:48:11.999865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:11.999773    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eacc871960c293b365a91e10e52ea23d90e33ea5b56d551e3ac7b008f23370e4"
Oct 30 19:48:12.001455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.001432    4925 generic.go:356] "Generic (PLEG): container finished" podID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerID="47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0" exitCode=0
Oct 30 19:48:12.001590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.001496    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"
Oct 30 19:48:12.001590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.001518    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" event={"ID":"95fd71d3-52ae-490a-96a2-3676b5a508bc","Type":"ContainerDied","Data":"47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0"}
Oct 30 19:48:12.001590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.001555    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz" event={"ID":"95fd71d3-52ae-490a-96a2-3676b5a508bc","Type":"ContainerDied","Data":"4ee3de7bb9869b756eece8fc837730e8154a0f86a933074508365a22c4c88153"}
Oct 30 19:48:12.003330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.003309    4925 generic.go:356] "Generic (PLEG): container finished" podID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerID="e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902" exitCode=0
Oct 30 19:48:12.003454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.003371    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"
Oct 30 19:48:12.003454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.003393    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" event={"ID":"94dc8df8-e90f-4697-aaa1-a332423097d2","Type":"ContainerDied","Data":"e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902"}
Oct 30 19:48:12.003454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.003449    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv" event={"ID":"94dc8df8-e90f-4697-aaa1-a332423097d2","Type":"ContainerDied","Data":"a50cdf149925be318acf0ca3551827d747851c2baad3ae42ed9c3416b084efb0"}
Oct 30 19:48:12.012474 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.012448    4925 scope.go:122] "RemoveContainer" containerID="a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546"
Oct 30 19:48:12.012917 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:12.012894    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546\": container with ID starting with a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546 not found: ID does not exist" containerID="a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546"
Oct 30 19:48:12.013031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.012931    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546"} err="failed to get container status \"a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546\": rpc error: code = NotFound desc = could not find container \"a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546\": container with ID starting with a5b87c5b5517b30aaa27817e3e185139ba52267aa139bdff9980350b103ea546 not found: ID does not exist"
Oct 30 19:48:12.013031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.012955    4925 scope.go:122] "RemoveContainer" containerID="47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0"
Oct 30 19:48:12.023944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.023925    4925 scope.go:122] "RemoveContainer" containerID="47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0"
Oct 30 19:48:12.024254 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:12.024236    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0\": container with ID starting with 47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0 not found: ID does not exist" containerID="47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0"
Oct 30 19:48:12.024350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.024265    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0"} err="failed to get container status \"47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0\": rpc error: code = NotFound desc = could not find container \"47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0\": container with ID starting with 47f88c4a484f0e804e3bb094e1a6242ef5a100811259be8b599e9d9698d474c0 not found: ID does not exist"
Oct 30 19:48:12.024350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.024289    4925 scope.go:122] "RemoveContainer" containerID="e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902"
Oct 30 19:48:12.035210 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.035181    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"]
Oct 30 19:48:12.035872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.035852    4925 scope.go:122] "RemoveContainer" containerID="e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902"
Oct 30 19:48:12.036169 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:12.036148    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902\": container with ID starting with e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902 not found: ID does not exist" containerID="e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902"
Oct 30 19:48:12.036232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.036179    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902"} err="failed to get container status \"e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902\": rpc error: code = NotFound desc = could not find container \"e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902\": container with ID starting with e6cb07b27bad36393ea3c89f5d6ac934670bed8bbada05f01978f29e769c0902 not found: ID does not exist"
Oct 30 19:48:12.041136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.041114    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8fe9f5-r6cw4"]
Oct 30 19:48:12.046045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.046020    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"]
Oct 30 19:48:12.053001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.052981    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-9gpf2"]
Oct 30 19:48:12.069520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.069495    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"]
Oct 30 19:48:12.072013 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.071992    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-p2jpz"]
Oct 30 19:48:12.083424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.083404    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"]
Oct 30 19:48:12.090483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.090463    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1b1531-openshift-default-56df9dbc55-8tnvv"]
Oct 30 19:48:12.564346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.564299    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:12.662445 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.662404    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"]
Oct 30 19:48:12.668974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.668949    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"]
Oct 30 19:48:12.830468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:12.830344    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:13.263944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:13.263911    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" path="/var/lib/kubelet/pods/2cdffc51-409a-4865-9be7-10e8d20e0882/volumes"
Oct 30 19:48:13.264389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:13.264374    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" path="/var/lib/kubelet/pods/94dc8df8-e90f-4697-aaa1-a332423097d2/volumes"
Oct 30 19:48:13.264836 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:13.264819    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" path="/var/lib/kubelet/pods/95fd71d3-52ae-490a-96a2-3676b5a508bc/volumes"
Oct 30 19:48:13.265540 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:13.265525    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc685e1c-aa9f-4851-9257-718a48a02781" path="/var/lib/kubelet/pods/dc685e1c-aa9f-4851-9257-718a48a02781/volumes"
Oct 30 19:48:13.447158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:13.447111    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:14.046649 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:14.046615    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:14.133592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:14.133559    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"]
Oct 30 19:48:14.137204 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:14.137167    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b281d5-jvfbk"]
Oct 30 19:48:14.225068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:14.225036    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:14.400098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:14.400068    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:14.738650 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:14.738567    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:15.263550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:15.263518    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e80a86-59c8-4c98-a056-dee68a94cc54" path="/var/lib/kubelet/pods/19e80a86-59c8-4c98-a056-dee68a94cc54/volumes"
Oct 30 19:48:15.422500 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:15.422452    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:15.827064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:15.827028    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk" podUID="6bd9fde7-3646-4e83-a958-9aad45c10daf" containerName="payload-processing" containerID="cri-o://60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78" gracePeriod=25
Oct 30 19:48:15.837727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:15.837698    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h" podUID="55bc197f-e609-4a15-a323-194cc2fbbf95" containerName="payload-pre-processing" containerID="cri-o://ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f" gracePeriod=25
Oct 30 19:48:16.533729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:16.533701    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:16.816704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:16.816601    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:16.990231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:16.988341    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:17.946813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:17.946779    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:18.121138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:18.121103    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:18.676535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:18.676318    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:19.195342 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195313    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"]
Oct 30 19:48:19.195870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195853    4925 cpu_manager.go:414] "containerMap: removing container" podUID="79867306-8d8e-44be-9f23-71ae7f9c5ad7" containerName="revoke-keys"
Oct 30 19:48:19.195923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195872    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79867306-8d8e-44be-9f23-71ae7f9c5ad7" containerName="revoke-keys"
Oct 30 19:48:19.195923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195896    4925 cpu_manager.go:414] "containerMap: removing container" podUID="32ba9701-eed9-402f-b28c-9ebe9b6e72ae" containerName="revoke-keys"
Oct 30 19:48:19.195923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195901    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ba9701-eed9-402f-b28c-9ebe9b6e72ae" containerName="revoke-keys"
Oct 30 19:48:19.195923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195921    4925 cpu_manager.go:414] "containerMap: removing container" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy"
Oct 30 19:48:19.196058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195927    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy"
Oct 30 19:48:19.196058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195943    4925 cpu_manager.go:414] "containerMap: removing container" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy"
Oct 30 19:48:19.196058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195948    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy"
Oct 30 19:48:19.196058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195955    4925 cpu_manager.go:414] "containerMap: removing container" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy"
Oct 30 19:48:19.196058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.195960    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy"
Oct 30 19:48:19.196058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.196034    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="79867306-8d8e-44be-9f23-71ae7f9c5ad7" containerName="revoke-keys"
Oct 30 19:48:19.196058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.196044    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ba9701-eed9-402f-b28c-9ebe9b6e72ae" containerName="revoke-keys"
Oct 30 19:48:19.196058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.196053    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="94dc8df8-e90f-4697-aaa1-a332423097d2" containerName="istio-proxy"
Oct 30 19:48:19.196301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.196065    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fd71d3-52ae-490a-96a2-3676b5a508bc" containerName="istio-proxy"
Oct 30 19:48:19.196301 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.196072    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdffc51-409a-4865-9be7-10e8d20e0882" containerName="istio-proxy"
Oct 30 19:48:19.201377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.201357    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.209985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.209942    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:19.210103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.210020    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:19.210103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.210079    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:19.210341 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.210309    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:19.210845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.210827    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"]
Oct 30 19:48:19.234222 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.234193    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"]
Oct 30 19:48:19.238250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.238221    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.251484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.251463    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkmdb\" (UniqueName: \"kubernetes.io/projected/96a9c464-b6b7-4a31-8338-96afc7cb9388-kube-api-access-tkmdb\") pod \"maas-api-revoke-sub-71362ddf-7pkcb\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.251618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.251494    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71362ddf-7pkcb\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.251618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.251521    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71362ddf-7pkcb\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.263008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.262985    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"]
Oct 30 19:48:19.352617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.352586    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ac1413b-kzjxz\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.352820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.352632    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdndj\" (UniqueName: \"kubernetes.io/projected/e6b9c38b-cbd5-437d-be46-605b76970539-kube-api-access-qdndj\") pod \"maas-api-revoke-sub-2ac1413b-kzjxz\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.352820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.352760    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkmdb\" (UniqueName: \"kubernetes.io/projected/96a9c464-b6b7-4a31-8338-96afc7cb9388-kube-api-access-tkmdb\") pod \"maas-api-revoke-sub-71362ddf-7pkcb\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.352820 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.352783    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ac1413b-kzjxz\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.353027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.352939    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71362ddf-7pkcb\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.353027 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.353010    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71362ddf-7pkcb\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.353907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.353841    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71362ddf-7pkcb\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.355789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.355763    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71362ddf-7pkcb\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.394840 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.394807    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkmdb\" (UniqueName: \"kubernetes.io/projected/96a9c464-b6b7-4a31-8338-96afc7cb9388-kube-api-access-tkmdb\") pod \"maas-api-revoke-sub-71362ddf-7pkcb\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.454313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.454251    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ac1413b-kzjxz\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.454440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.454312    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdndj\" (UniqueName: \"kubernetes.io/projected/e6b9c38b-cbd5-437d-be46-605b76970539-kube-api-access-qdndj\") pod \"maas-api-revoke-sub-2ac1413b-kzjxz\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.454440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.454353    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ac1413b-kzjxz\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.455015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.454994    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ac1413b-kzjxz\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.456701 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.456665    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ac1413b-kzjxz\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.466291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.466268    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdndj\" (UniqueName: \"kubernetes.io/projected/e6b9c38b-cbd5-437d-be46-605b76970539-kube-api-access-qdndj\") pod \"maas-api-revoke-sub-2ac1413b-kzjxz\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.517272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.517248    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:19.551063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.551035    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:19.736873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.736793    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:19.802899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.802873    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"]
Oct 30 19:48:19.805938 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:48:19.805905    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a9c464_b6b7_4a31_8338_96afc7cb9388.slice/crio-3a7b2f3c34244a3792e93d4e376ed42e769e06f7e5b9825fc865de3f4a0175f9 WatchSource:0}: Error finding container 3a7b2f3c34244a3792e93d4e376ed42e769e06f7e5b9825fc865de3f4a0175f9: Status 404 returned error can't find the container with id 3a7b2f3c34244a3792e93d4e376ed42e769e06f7e5b9825fc865de3f4a0175f9
Oct 30 19:48:19.856015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:19.855988    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"]
Oct 30 19:48:19.857642 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:48:19.857616    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b9c38b_cbd5_437d_be46_605b76970539.slice/crio-5e32ff7a095dd673d58c0491dc0c60a11fcc5fc5faa1183e48fe71beb1480a03 WatchSource:0}: Error finding container 5e32ff7a095dd673d58c0491dc0c60a11fcc5fc5faa1183e48fe71beb1480a03: Status 404 returned error can't find the container with id 5e32ff7a095dd673d58c0491dc0c60a11fcc5fc5faa1183e48fe71beb1480a03
Oct 30 19:48:20.067603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:20.067566    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb" event={"ID":"96a9c464-b6b7-4a31-8338-96afc7cb9388","Type":"ContainerStarted","Data":"3a7b2f3c34244a3792e93d4e376ed42e769e06f7e5b9825fc865de3f4a0175f9"}
Oct 30 19:48:20.068951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:20.068926    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz" event={"ID":"e6b9c38b-cbd5-437d-be46-605b76970539","Type":"ContainerStarted","Data":"5e32ff7a095dd673d58c0491dc0c60a11fcc5fc5faa1183e48fe71beb1480a03"}
Oct 30 19:48:20.458146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:20.458108    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:21.065196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:21.062384    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:21.079186 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:21.079155    4925 generic.go:356] "Generic (PLEG): container finished" podID="96a9c464-b6b7-4a31-8338-96afc7cb9388" containerID="b5f56054f5cb9872faa95a3d1fcf3898f08439a8982ff9e027d9673b303d0457" exitCode=0
Oct 30 19:48:21.079378 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:21.079241    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb" event={"ID":"96a9c464-b6b7-4a31-8338-96afc7cb9388","Type":"ContainerDied","Data":"b5f56054f5cb9872faa95a3d1fcf3898f08439a8982ff9e027d9673b303d0457"}
Oct 30 19:48:21.080880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:21.080846    4925 generic.go:356] "Generic (PLEG): container finished" podID="e6b9c38b-cbd5-437d-be46-605b76970539" containerID="db86b5c98db2b4c454aa8b0055d4db86023be3d18ae3c03a694cb244d1645c34" exitCode=0
Oct 30 19:48:21.080983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:21.080924    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz" event={"ID":"e6b9c38b-cbd5-437d-be46-605b76970539","Type":"ContainerDied","Data":"db86b5c98db2b4c454aa8b0055d4db86023be3d18ae3c03a694cb244d1645c34"}
Oct 30 19:48:21.246154 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:21.246120    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:21.998711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:21.993020    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:22.041934 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.040644    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"]
Oct 30 19:48:22.046094 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.046067    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18d680f-sjhll"]
Oct 30 19:48:22.133667 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:22.133627    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:48:22.278151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.278127    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:22.281382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.281363    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:22.295430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.295403    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b9c38b-cbd5-437d-be46-605b76970539-kube-api-access-qdndj\" (UniqueName: \"kubernetes.io/projected/e6b9c38b-cbd5-437d-be46-605b76970539-kube-api-access-qdndj\") pod \"e6b9c38b-cbd5-437d-be46-605b76970539\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") "
Oct 30 19:48:22.295547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.295479    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-openshift-ca\") pod \"e6b9c38b-cbd5-437d-be46-605b76970539\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") "
Oct 30 19:48:22.295547 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.295511    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-xks-ca\") pod \"e6b9c38b-cbd5-437d-be46-605b76970539\" (UID: \"e6b9c38b-cbd5-437d-be46-605b76970539\") "
Oct 30 19:48:22.297418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.297056    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-openshift-ca" pod "e6b9c38b-cbd5-437d-be46-605b76970539" (UID: "e6b9c38b-cbd5-437d-be46-605b76970539"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:22.299377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.298824    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-xks-ca" pod "e6b9c38b-cbd5-437d-be46-605b76970539" (UID: "e6b9c38b-cbd5-437d-be46-605b76970539"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:22.299755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.299580    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b9c38b-cbd5-437d-be46-605b76970539-kube-api-access-qdndj" pod "e6b9c38b-cbd5-437d-be46-605b76970539" (UID: "e6b9c38b-cbd5-437d-be46-605b76970539"). InnerVolumeSpecName "kube-api-access-qdndj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:22.397575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.397544    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-xks-ca\") pod \"96a9c464-b6b7-4a31-8338-96afc7cb9388\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") "
Oct 30 19:48:22.397830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.397603    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a9c464-b6b7-4a31-8338-96afc7cb9388-kube-api-access-tkmdb\" (UniqueName: \"kubernetes.io/projected/96a9c464-b6b7-4a31-8338-96afc7cb9388-kube-api-access-tkmdb\") pod \"96a9c464-b6b7-4a31-8338-96afc7cb9388\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") "
Oct 30 19:48:22.397830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.397665    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-openshift-ca\") pod \"96a9c464-b6b7-4a31-8338-96afc7cb9388\" (UID: \"96a9c464-b6b7-4a31-8338-96afc7cb9388\") "
Oct 30 19:48:22.398128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.398104    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-openshift-ca" pod "96a9c464-b6b7-4a31-8338-96afc7cb9388" (UID: "96a9c464-b6b7-4a31-8338-96afc7cb9388"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:22.398226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.398128    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdndj\" (UniqueName: \"kubernetes.io/projected/e6b9c38b-cbd5-437d-be46-605b76970539-kube-api-access-qdndj\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.398226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.398150    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.398226 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.398164    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b9c38b-cbd5-437d-be46-605b76970539-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.400052 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.400025    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-xks-ca" pod "96a9c464-b6b7-4a31-8338-96afc7cb9388" (UID: "96a9c464-b6b7-4a31-8338-96afc7cb9388"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:22.400131 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.400112    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a9c464-b6b7-4a31-8338-96afc7cb9388-kube-api-access-tkmdb" pod "96a9c464-b6b7-4a31-8338-96afc7cb9388" (UID: "96a9c464-b6b7-4a31-8338-96afc7cb9388"). InnerVolumeSpecName "kube-api-access-tkmdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:22.499501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.499475    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.499501 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.499500    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkmdb\" (UniqueName: \"kubernetes.io/projected/96a9c464-b6b7-4a31-8338-96afc7cb9388-kube-api-access-tkmdb\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.499714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.499510    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a9c464-b6b7-4a31-8338-96afc7cb9388-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:22.738113 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.738077    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"]
Oct 30 19:48:22.738369 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.738341    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj" podUID="ce66c736-8ac9-4ff3-8f0f-19204795da0d" containerName="payload-processing" containerID="cri-o://6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab" gracePeriod=30
Oct 30 19:48:22.776898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.776866    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:22.964223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.964187    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:22.995572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:22.995517    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:48:23.096881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.096851    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"
Oct 30 19:48:23.096881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.096864    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz" event={"ID":"e6b9c38b-cbd5-437d-be46-605b76970539","Type":"ContainerDied","Data":"5e32ff7a095dd673d58c0491dc0c60a11fcc5fc5faa1183e48fe71beb1480a03"}
Oct 30 19:48:23.097403 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.096917    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e32ff7a095dd673d58c0491dc0c60a11fcc5fc5faa1183e48fe71beb1480a03"
Oct 30 19:48:23.098483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.098457    4925 generic.go:356] "Generic (PLEG): container finished" podID="ce66c736-8ac9-4ff3-8f0f-19204795da0d" containerID="6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab" exitCode=0
Oct 30 19:48:23.098635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.098484    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj" event={"ID":"ce66c736-8ac9-4ff3-8f0f-19204795da0d","Type":"ContainerDied","Data":"6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab"}
Oct 30 19:48:23.098635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.098517    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"
Oct 30 19:48:23.098635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.098536    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj" event={"ID":"ce66c736-8ac9-4ff3-8f0f-19204795da0d","Type":"ContainerDied","Data":"ed5886e50aaf04dc683c3ae45e208312e5f1809923e271c977f0751201f922db"}
Oct 30 19:48:23.098635 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.098567    4925 scope.go:122] "RemoveContainer" containerID="6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab"
Oct 30 19:48:23.100401 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.100375    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb" event={"ID":"96a9c464-b6b7-4a31-8338-96afc7cb9388","Type":"ContainerDied","Data":"3a7b2f3c34244a3792e93d4e376ed42e769e06f7e5b9825fc865de3f4a0175f9"}
Oct 30 19:48:23.100541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.100410    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"
Oct 30 19:48:23.100541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.100418    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a7b2f3c34244a3792e93d4e376ed42e769e06f7e5b9825fc865de3f4a0175f9"
Oct 30 19:48:23.105841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.105813    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce66c736-8ac9-4ff3-8f0f-19204795da0d-kube-api-access-6sfnx\" (UniqueName: \"kubernetes.io/projected/ce66c736-8ac9-4ff3-8f0f-19204795da0d-kube-api-access-6sfnx\") pod \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\" (UID: \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\") "
Oct 30 19:48:23.105964 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.105908    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce66c736-8ac9-4ff3-8f0f-19204795da0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66c736-8ac9-4ff3-8f0f-19204795da0d-plugins-config-volume\") pod \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\" (UID: \"ce66c736-8ac9-4ff3-8f0f-19204795da0d\") "
Oct 30 19:48:23.106492 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.106463    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce66c736-8ac9-4ff3-8f0f-19204795da0d-plugins-config-volume" pod "ce66c736-8ac9-4ff3-8f0f-19204795da0d" (UID: "ce66c736-8ac9-4ff3-8f0f-19204795da0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:23.106995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.106970    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66c736-8ac9-4ff3-8f0f-19204795da0d-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:23.108869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.108847    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce66c736-8ac9-4ff3-8f0f-19204795da0d-kube-api-access-6sfnx" pod "ce66c736-8ac9-4ff3-8f0f-19204795da0d" (UID: "ce66c736-8ac9-4ff3-8f0f-19204795da0d"). InnerVolumeSpecName "kube-api-access-6sfnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:23.111530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.111515    4925 scope.go:122] "RemoveContainer" containerID="6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab"
Oct 30 19:48:23.111915 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:23.111895    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab\": container with ID starting with 6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab not found: ID does not exist" containerID="6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab"
Oct 30 19:48:23.111981 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.111924    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab"} err="failed to get container status \"6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab\": rpc error: code = NotFound desc = could not find container \"6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab\": container with ID starting with 6be5b132e14028609da22cf8693791a654996298798ca036a51571128fb1c2ab not found: ID does not exist"
Oct 30 19:48:23.133600 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.133576    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:23.208555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.208520    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfnx\" (UniqueName: \"kubernetes.io/projected/ce66c736-8ac9-4ff3-8f0f-19204795da0d-kube-api-access-6sfnx\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:23.341076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.338732    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:23.348712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.344773    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64441e28-10bb-40af-8ff0-5eb837675923" path="/var/lib/kubelet/pods/64441e28-10bb-40af-8ff0-5eb837675923/volumes"
Oct 30 19:48:23.348712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.345462    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"]
Oct 30 19:48:23.348712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.345728    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy" containerID="cri-o://e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3" gracePeriod=30
Oct 30 19:48:23.462923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.462869    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"]
Oct 30 19:48:23.471777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.471734    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-be6b74-6dbc7d9ddd-pmhnj"]
Oct 30 19:48:23.513890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.513796    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:23.674891 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:23.674856    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:24.034732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:24.029339    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:24.773355 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:24.773315    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:25.192727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:25.192671    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h" podUID="55bc197f-e609-4a15-a323-194cc2fbbf95" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:25.192910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:25.192806    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk" podUID="6bd9fde7-3646-4e83-a958-9aad45c10daf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:25.262356 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:25.262320    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce66c736-8ac9-4ff3-8f0f-19204795da0d" path="/var/lib/kubelet/pods/ce66c736-8ac9-4ff3-8f0f-19204795da0d/volumes"
Oct 30 19:48:25.743590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:25.743551    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:26.367541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:26.367507    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:27.029424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.029383    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main" containerID="cri-o://586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa" gracePeriod=45
Oct 30 19:48:27.158291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.157776    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:27.336075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.336053    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:48:27.408099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.408062    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-dshm\") pod \"e9f27b3d-570a-448e-8926-ed6643d8903b\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") "
Oct 30 19:48:27.408550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.408135    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-home\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-home\") pod \"e9f27b3d-570a-448e-8926-ed6643d8903b\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") "
Oct 30 19:48:27.408550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.408177    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-model-cache\") pod \"e9f27b3d-570a-448e-8926-ed6643d8903b\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") "
Oct 30 19:48:27.408550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.408246    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f27b3d-570a-448e-8926-ed6643d8903b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f27b3d-570a-448e-8926-ed6643d8903b-tls-certs\") pod \"e9f27b3d-570a-448e-8926-ed6643d8903b\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") "
Oct 30 19:48:27.408550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.408407    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-home" pod "e9f27b3d-570a-448e-8926-ed6643d8903b" (UID: "e9f27b3d-570a-448e-8926-ed6643d8903b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:27.408550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.408481    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f27b3d-570a-448e-8926-ed6643d8903b-kube-api-access-228kq\" (UniqueName: \"kubernetes.io/projected/e9f27b3d-570a-448e-8926-ed6643d8903b-kube-api-access-228kq\") pod \"e9f27b3d-570a-448e-8926-ed6643d8903b\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") "
Oct 30 19:48:27.408550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.408516    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-tmp-dir\") pod \"e9f27b3d-570a-448e-8926-ed6643d8903b\" (UID: \"e9f27b3d-570a-448e-8926-ed6643d8903b\") "
Oct 30 19:48:27.408933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.408482    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-model-cache" pod "e9f27b3d-570a-448e-8926-ed6643d8903b" (UID: "e9f27b3d-570a-448e-8926-ed6643d8903b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:27.409053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.409028    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.409053 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.409037    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-tmp-dir" pod "e9f27b3d-570a-448e-8926-ed6643d8903b" (UID: "e9f27b3d-570a-448e-8926-ed6643d8903b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:27.409146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.409049    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.410561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.410533    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-dshm" pod "e9f27b3d-570a-448e-8926-ed6643d8903b" (UID: "e9f27b3d-570a-448e-8926-ed6643d8903b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:27.410826 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.410808    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f27b3d-570a-448e-8926-ed6643d8903b-tls-certs" pod "e9f27b3d-570a-448e-8926-ed6643d8903b" (UID: "e9f27b3d-570a-448e-8926-ed6643d8903b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:27.410982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.410968    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f27b3d-570a-448e-8926-ed6643d8903b-kube-api-access-228kq" pod "e9f27b3d-570a-448e-8926-ed6643d8903b" (UID: "e9f27b3d-570a-448e-8926-ed6643d8903b"). InnerVolumeSpecName "kube-api-access-228kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:27.510030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.509999    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-228kq\" (UniqueName: \"kubernetes.io/projected/e9f27b3d-570a-448e-8926-ed6643d8903b-kube-api-access-228kq\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.510030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.510027    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.510030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.510038    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f27b3d-570a-448e-8926-ed6643d8903b-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.510285 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.510046    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f27b3d-570a-448e-8926-ed6643d8903b-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:27.685129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.685090    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" containerID="cri-o://345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853" gracePeriod=45
Oct 30 19:48:27.689678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.689640    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main" containerID="cri-o://7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7" gracePeriod=45
Oct 30 19:48:27.762679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.762645    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:27.812424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.812050    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": dial tcp 10.132.0.54:8000: connect: connection refused"
Oct 30 19:48:27.967254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.967229    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:48:27.973661 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:27.973430    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:48:28.015381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.015344    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a520025-69b4-4288-b951-0333b96ed430-kube-api-access-tq9cr\" (UniqueName: \"kubernetes.io/projected/0a520025-69b4-4288-b951-0333b96ed430-kube-api-access-tq9cr\") pod \"0a520025-69b4-4288-b951-0333b96ed430\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") "
Oct 30 19:48:28.015575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.015403    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-home\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-home\") pod \"0a520025-69b4-4288-b951-0333b96ed430\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") "
Oct 30 19:48:28.015575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.015470    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-model-cache\") pod \"0a520025-69b4-4288-b951-0333b96ed430\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") "
Oct 30 19:48:28.015575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.015523    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a520025-69b4-4288-b951-0333b96ed430-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a520025-69b4-4288-b951-0333b96ed430-tls-certs\") pod \"0a520025-69b4-4288-b951-0333b96ed430\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") "
Oct 30 19:48:28.015799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.015583    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-dshm\") pod \"0a520025-69b4-4288-b951-0333b96ed430\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") "
Oct 30 19:48:28.015799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.015785    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-tmp-dir\") pod \"0a520025-69b4-4288-b951-0333b96ed430\" (UID: \"0a520025-69b4-4288-b951-0333b96ed430\") "
Oct 30 19:48:28.016137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.016107    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-model-cache" pod "0a520025-69b4-4288-b951-0333b96ed430" (UID: "0a520025-69b4-4288-b951-0333b96ed430"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.016528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.016311    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-home" pod "0a520025-69b4-4288-b951-0333b96ed430" (UID: "0a520025-69b4-4288-b951-0333b96ed430"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.016528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.016336    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-tmp-dir" pod "0a520025-69b4-4288-b951-0333b96ed430" (UID: "0a520025-69b4-4288-b951-0333b96ed430"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.016528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.016476    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.016528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.016494    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.016528 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.016507    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.018823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.018800    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-dshm" pod "0a520025-69b4-4288-b951-0333b96ed430" (UID: "0a520025-69b4-4288-b951-0333b96ed430"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.019072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.019053    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a520025-69b4-4288-b951-0333b96ed430-kube-api-access-tq9cr" pod "0a520025-69b4-4288-b951-0333b96ed430" (UID: "0a520025-69b4-4288-b951-0333b96ed430"). InnerVolumeSpecName "kube-api-access-tq9cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:28.021062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.021037    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a520025-69b4-4288-b951-0333b96ed430-tls-certs" pod "0a520025-69b4-4288-b951-0333b96ed430" (UID: "0a520025-69b4-4288-b951-0333b96ed430"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:28.117670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.117108    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234d689f-2905-4f14-b56d-2906a7d4667c-kube-api-access-jcc4j\" (UniqueName: \"kubernetes.io/projected/234d689f-2905-4f14-b56d-2906a7d4667c-kube-api-access-jcc4j\") pod \"234d689f-2905-4f14-b56d-2906a7d4667c\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") "
Oct 30 19:48:28.117670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.117160    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-home\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-home\") pod \"234d689f-2905-4f14-b56d-2906a7d4667c\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") "
Oct 30 19:48:28.117670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.117221    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-tmp-dir\") pod \"234d689f-2905-4f14-b56d-2906a7d4667c\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") "
Oct 30 19:48:28.117670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.117299    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-model-cache\") pod \"234d689f-2905-4f14-b56d-2906a7d4667c\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") "
Oct 30 19:48:28.117670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.117554    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234d689f-2905-4f14-b56d-2906a7d4667c-tls-certs\" (UniqueName: \"kubernetes.io/secret/234d689f-2905-4f14-b56d-2906a7d4667c-tls-certs\") pod \"234d689f-2905-4f14-b56d-2906a7d4667c\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") "
Oct 30 19:48:28.117670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.117585    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-dshm\") pod \"234d689f-2905-4f14-b56d-2906a7d4667c\" (UID: \"234d689f-2905-4f14-b56d-2906a7d4667c\") "
Oct 30 19:48:28.118550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.118397    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-tmp-dir" pod "234d689f-2905-4f14-b56d-2906a7d4667c" (UID: "234d689f-2905-4f14-b56d-2906a7d4667c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.118936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.118911    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9cr\" (UniqueName: \"kubernetes.io/projected/0a520025-69b4-4288-b951-0333b96ed430-kube-api-access-tq9cr\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.120755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.118938    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.120755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.118953    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a520025-69b4-4288-b951-0333b96ed430-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.120755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.118966    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a520025-69b4-4288-b951-0333b96ed430-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.120755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.119196    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-model-cache" pod "234d689f-2905-4f14-b56d-2906a7d4667c" (UID: "234d689f-2905-4f14-b56d-2906a7d4667c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.120755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.119507    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-home" pod "234d689f-2905-4f14-b56d-2906a7d4667c" (UID: "234d689f-2905-4f14-b56d-2906a7d4667c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.122470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.122450    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-dshm" pod "234d689f-2905-4f14-b56d-2906a7d4667c" (UID: "234d689f-2905-4f14-b56d-2906a7d4667c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.122937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.122919    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234d689f-2905-4f14-b56d-2906a7d4667c-tls-certs" pod "234d689f-2905-4f14-b56d-2906a7d4667c" (UID: "234d689f-2905-4f14-b56d-2906a7d4667c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:28.123308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.123284    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234d689f-2905-4f14-b56d-2906a7d4667c-kube-api-access-jcc4j" pod "234d689f-2905-4f14-b56d-2906a7d4667c" (UID: "234d689f-2905-4f14-b56d-2906a7d4667c"). InnerVolumeSpecName "kube-api-access-jcc4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:28.144968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.144936    4925 generic.go:356] "Generic (PLEG): container finished" podID="234d689f-2905-4f14-b56d-2906a7d4667c" containerID="7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7" exitCode=0
Oct 30 19:48:28.145093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.145059    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"
Oct 30 19:48:28.145163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.145149    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" event={"ID":"234d689f-2905-4f14-b56d-2906a7d4667c","Type":"ContainerDied","Data":"7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7"}
Oct 30 19:48:28.145258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.145186    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t" event={"ID":"234d689f-2905-4f14-b56d-2906a7d4667c","Type":"ContainerDied","Data":"5fe0221d07941ec45e8c364442633661b6d0840d2a3574dc0d4ec7b6ee17dd4d"}
Oct 30 19:48:28.145258 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.145209    4925 scope.go:122] "RemoveContainer" containerID="7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7"
Oct 30 19:48:28.146855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.146768    4925 generic.go:356] "Generic (PLEG): container finished" podID="0a520025-69b4-4288-b951-0333b96ed430" containerID="345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853" exitCode=0
Oct 30 19:48:28.146855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.146830    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"
Oct 30 19:48:28.146999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.146861    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" event={"ID":"0a520025-69b4-4288-b951-0333b96ed430","Type":"ContainerDied","Data":"345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853"}
Oct 30 19:48:28.146999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.146902    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk" event={"ID":"0a520025-69b4-4288-b951-0333b96ed430","Type":"ContainerDied","Data":"8f294be4160b2e73894f1c491c91e208ecd6bf25626cabe9b456bd25476f86bb"}
Oct 30 19:48:28.148463 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.148423    4925 generic.go:356] "Generic (PLEG): container finished" podID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerID="586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa" exitCode=0
Oct 30 19:48:28.148628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.148494    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" event={"ID":"e9f27b3d-570a-448e-8926-ed6643d8903b","Type":"ContainerDied","Data":"586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa"}
Oct 30 19:48:28.148628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.148519    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj" event={"ID":"e9f27b3d-570a-448e-8926-ed6643d8903b","Type":"ContainerDied","Data":"c506c9691b901ac70f01076c23f983ffe4a894ee9c6264f0d602199b7d49f0fb"}
Oct 30 19:48:28.148841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.148651    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"
Oct 30 19:48:28.162432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.162411    4925 scope.go:122] "RemoveContainer" containerID="7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7"
Oct 30 19:48:28.163279 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:28.163158    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7\": container with ID starting with 7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7 not found: ID does not exist" containerID="7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7"
Oct 30 19:48:28.163279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.163195    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7"} err="failed to get container status \"7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7\": rpc error: code = NotFound desc = could not find container \"7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7\": container with ID starting with 7c822ce41b5cb83efda189d25ca6d8f7855d2a39ed10eeef5c783e38e23faee7 not found: ID does not exist"
Oct 30 19:48:28.163279 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.163218    4925 scope.go:122] "RemoveContainer" containerID="345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853"
Oct 30 19:48:28.173196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.173174    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"]
Oct 30 19:48:28.173723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.173675    4925 scope.go:122] "RemoveContainer" containerID="345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853"
Oct 30 19:48:28.173985 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:28.173969    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853\": container with ID starting with 345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853 not found: ID does not exist" containerID="345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853"
Oct 30 19:48:28.174043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.173990    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853"} err="failed to get container status \"345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853\": rpc error: code = NotFound desc = could not find container \"345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853\": container with ID starting with 345c1e78e3f0b77a10945caa521bb161e4f84b33abc309f75866052ff02a9853 not found: ID does not exist"
Oct 30 19:48:28.174043 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.174004    4925 scope.go:122] "RemoveContainer" containerID="586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa"
Oct 30 19:48:28.179985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.179966    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/simulator-w5-2d5d53-kserve-7dc86d6dbf-cfb4t"]
Oct 30 19:48:28.184073 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.184059    4925 scope.go:122] "RemoveContainer" containerID="586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa"
Oct 30 19:48:28.184328 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:28.184310    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa\": container with ID starting with 586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa not found: ID does not exist" containerID="586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa"
Oct 30 19:48:28.184385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.184333    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa"} err="failed to get container status \"586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa\": rpc error: code = NotFound desc = could not find container \"586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa\": container with ID starting with 586eb5d6ba3fd5dff5af9e539b1a16a9da3558f2b466500cfa3f7e48801074fa not found: ID does not exist"
Oct 30 19:48:28.189351 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.189333    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"]
Oct 30 19:48:28.192755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.192736    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2d5d53/premium-w5-2d5d53-kserve-5549c5ff96-488bk"]
Oct 30 19:48:28.205716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.205649    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"]
Oct 30 19:48:28.208067 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.208050    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-27b5d5/simulator-w4-27b5d5-kserve-8685bd4ccb-krvfj"]
Oct 30 19:48:28.220291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.220273    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcc4j\" (UniqueName: \"kubernetes.io/projected/234d689f-2905-4f14-b56d-2906a7d4667c-kube-api-access-jcc4j\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.220360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.220294    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.220360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.220304    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.220360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.220313    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234d689f-2905-4f14-b56d-2906a7d4667c-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.220360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.220320    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234d689f-2905-4f14-b56d-2906a7d4667c-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.424508 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.424349    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:28.629287 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.629262    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:48:28.733848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.733771    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-credential-socket\") pod \"f8313268-9549-42a2-942b-aac4e43c75a3\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") "
Oct 30 19:48:28.733848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.733810    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8313268-9549-42a2-942b-aac4e43c75a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8313268-9549-42a2-942b-aac4e43c75a3-istio-podinfo\") pod \"f8313268-9549-42a2-942b-aac4e43c75a3\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") "
Oct 30 19:48:28.733848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.733840    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-socket\") pod \"f8313268-9549-42a2-942b-aac4e43c75a3\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") "
Oct 30 19:48:28.734152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.733877    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-data\") pod \"f8313268-9549-42a2-942b-aac4e43c75a3\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") "
Oct 30 19:48:28.734152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.733954    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-kube-api-access-497d6\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-kube-api-access-497d6\") pod \"f8313268-9549-42a2-942b-aac4e43c75a3\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") "
Oct 30 19:48:28.734152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.733978    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-envoy\") pod \"f8313268-9549-42a2-942b-aac4e43c75a3\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") "
Oct 30 19:48:28.734152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734006    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-certs\") pod \"f8313268-9549-42a2-942b-aac4e43c75a3\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") "
Oct 30 19:48:28.734152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734046    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-credential-socket" pod "f8313268-9549-42a2-942b-aac4e43c75a3" (UID: "f8313268-9549-42a2-942b-aac4e43c75a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.734152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734084    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-istio-token\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-istio-token\") pod \"f8313268-9549-42a2-942b-aac4e43c75a3\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") "
Oct 30 19:48:28.734152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734128    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8313268-9549-42a2-942b-aac4e43c75a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8313268-9549-42a2-942b-aac4e43c75a3-istiod-ca-cert\") pod \"f8313268-9549-42a2-942b-aac4e43c75a3\" (UID: \"f8313268-9549-42a2-942b-aac4e43c75a3\") "
Oct 30 19:48:28.734552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734150    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-socket" pod "f8313268-9549-42a2-942b-aac4e43c75a3" (UID: "f8313268-9549-42a2-942b-aac4e43c75a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.734552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734189    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-data" pod "f8313268-9549-42a2-942b-aac4e43c75a3" (UID: "f8313268-9549-42a2-942b-aac4e43c75a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.734552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734375    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-certs" pod "f8313268-9549-42a2-942b-aac4e43c75a3" (UID: "f8313268-9549-42a2-942b-aac4e43c75a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.734552 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734497    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8313268-9549-42a2-942b-aac4e43c75a3-istiod-ca-cert" pod "f8313268-9549-42a2-942b-aac4e43c75a3" (UID: "f8313268-9549-42a2-942b-aac4e43c75a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:28.734968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734636    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.734968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734656    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8313268-9549-42a2-942b-aac4e43c75a3-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.734968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734671    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.734968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734716    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.734968 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.734730    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.736350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.736322    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8313268-9549-42a2-942b-aac4e43c75a3-istio-podinfo" pod "f8313268-9549-42a2-942b-aac4e43c75a3" (UID: "f8313268-9549-42a2-942b-aac4e43c75a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:28.736468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.736359    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-envoy" pod "f8313268-9549-42a2-942b-aac4e43c75a3" (UID: "f8313268-9549-42a2-942b-aac4e43c75a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:28.736468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.736369    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-kube-api-access-497d6" pod "f8313268-9549-42a2-942b-aac4e43c75a3" (UID: "f8313268-9549-42a2-942b-aac4e43c75a3"). InnerVolumeSpecName "kube-api-access-497d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:28.736543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.736506    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-istio-token" pod "f8313268-9549-42a2-942b-aac4e43c75a3" (UID: "f8313268-9549-42a2-942b-aac4e43c75a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:28.836247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.836217    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8313268-9549-42a2-942b-aac4e43c75a3-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.836247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.836240    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-497d6\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-kube-api-access-497d6\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.836247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.836251    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8313268-9549-42a2-942b-aac4e43c75a3-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:28.836435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:28.836259    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8313268-9549-42a2-942b-aac4e43c75a3-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.161092 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.161056    4925 generic.go:356] "Generic (PLEG): container finished" podID="f8313268-9549-42a2-942b-aac4e43c75a3" containerID="e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3" exitCode=0
Oct 30 19:48:29.161303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.161103    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"
Oct 30 19:48:29.161303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.161139    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" event={"ID":"f8313268-9549-42a2-942b-aac4e43c75a3","Type":"ContainerDied","Data":"e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3"}
Oct 30 19:48:29.161303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.161234    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs" event={"ID":"f8313268-9549-42a2-942b-aac4e43c75a3","Type":"ContainerDied","Data":"61945daa661c6836ade573195eac44f6988cb14a0d90b3b2119b1953132ad66b"}
Oct 30 19:48:29.161303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.161265    4925 scope.go:122] "RemoveContainer" containerID="e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3"
Oct 30 19:48:29.173044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.173026    4925 scope.go:122] "RemoveContainer" containerID="28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1"
Oct 30 19:48:29.182895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.182875    4925 scope.go:122] "RemoveContainer" containerID="e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3"
Oct 30 19:48:29.183190 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:29.183166    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3\": container with ID starting with e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3 not found: ID does not exist" containerID="e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3"
Oct 30 19:48:29.183282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.183196    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3"} err="failed to get container status \"e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3\": rpc error: code = NotFound desc = could not find container \"e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3\": container with ID starting with e7f19a147a6c39ddfdcba5387d7143956e10e0f9820c90f72e0064d6d1d19af3 not found: ID does not exist"
Oct 30 19:48:29.183282 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.183214    4925 scope.go:122] "RemoveContainer" containerID="28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1"
Oct 30 19:48:29.183502 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:29.183486    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1\": container with ID starting with 28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1 not found: ID does not exist" containerID="28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1"
Oct 30 19:48:29.183545 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.183508    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1"} err="failed to get container status \"28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1\": rpc error: code = NotFound desc = could not find container \"28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1\": container with ID starting with 28d2dd14a6c7c099e4b4ae053d23157b387ee6aea99eae0d67b8fad239be41a1 not found: ID does not exist"
Oct 30 19:48:29.184146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.184128    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"]
Oct 30 19:48:29.193129 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.193105    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b281d5-openshift-default-fcfb8bfc5-5rncs"]
Oct 30 19:48:29.264022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.263995    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a520025-69b4-4288-b951-0333b96ed430" path="/var/lib/kubelet/pods/0a520025-69b4-4288-b951-0333b96ed430/volumes"
Oct 30 19:48:29.264407 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.264394    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" path="/var/lib/kubelet/pods/234d689f-2905-4f14-b56d-2906a7d4667c/volumes"
Oct 30 19:48:29.264814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.264800    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" path="/var/lib/kubelet/pods/e9f27b3d-570a-448e-8926-ed6643d8903b/volumes"
Oct 30 19:48:29.265487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.265473    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" path="/var/lib/kubelet/pods/f8313268-9549-42a2-942b-aac4e43c75a3/volumes"
Oct 30 19:48:29.364187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.364004    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:29.954577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:29.954541    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:30.759708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:30.759662    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:31.276314 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:31.276277    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:31.446809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:31.446776    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:31.516619 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:31.516586    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"]
Oct 30 19:48:31.516862 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:31.516839    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy" containerID="cri-o://205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8" gracePeriod=30
Oct 30 19:48:31.627804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:31.626856    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:32.015494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.015463    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"]
Oct 30 19:48:32.015944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.015929    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main"
Oct 30 19:48:32.015944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.015944    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.015957    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.015961    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.015975    4925 cpu_manager.go:414] "containerMap: removing container" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.015980    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.015994    4925 cpu_manager.go:414] "containerMap: removing container" podUID="96a9c464-b6b7-4a31-8338-96afc7cb9388" containerName="revoke-keys"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016000    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a9c464-b6b7-4a31-8338-96afc7cb9388" containerName="revoke-keys"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016009    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016013    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016023    4925 cpu_manager.go:414] "containerMap: removing container" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016027    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016042    4925 cpu_manager.go:414] "containerMap: removing container" podUID="ce66c736-8ac9-4ff3-8f0f-19204795da0d" containerName="payload-processing"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016047    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce66c736-8ac9-4ff3-8f0f-19204795da0d" containerName="payload-processing"
Oct 30 19:48:32.016047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016053    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e6b9c38b-cbd5-437d-be46-605b76970539" containerName="revoke-keys"
Oct 30 19:48:32.016486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016058    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b9c38b-cbd5-437d-be46-605b76970539" containerName="revoke-keys"
Oct 30 19:48:32.016486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016129    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a520025-69b4-4288-b951-0333b96ed430" containerName="main"
Oct 30 19:48:32.016486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016140    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a9c464-b6b7-4a31-8338-96afc7cb9388" containerName="revoke-keys"
Oct 30 19:48:32.016486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016147    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy"
Oct 30 19:48:32.016486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016153    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="234d689f-2905-4f14-b56d-2906a7d4667c" containerName="main"
Oct 30 19:48:32.016486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016160    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce66c736-8ac9-4ff3-8f0f-19204795da0d" containerName="payload-processing"
Oct 30 19:48:32.016486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016165    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b9c38b-cbd5-437d-be46-605b76970539" containerName="revoke-keys"
Oct 30 19:48:32.016486 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.016171    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f27b3d-570a-448e-8926-ed6643d8903b" containerName="main"
Oct 30 19:48:32.019251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.019235    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.025668 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.025640    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:32.025845 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.025752    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:32.029830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.029417    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"]
Oct 30 19:48:32.098085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.098057    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.098231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.098121    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.098294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.098279    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74255\" (UniqueName: \"kubernetes.io/projected/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-kube-api-access-74255\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.120143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.120118    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:32.120251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.120172    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:32.199879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.199847    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.200096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.199941    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.200096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.200046    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74255\" (UniqueName: \"kubernetes.io/projected/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-kube-api-access-74255\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.200624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.200603    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.202456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.202430    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.212645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.212483    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74255\" (UniqueName: \"kubernetes.io/projected/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-kube-api-access-74255\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.335039 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.334933    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:32.342527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.342497    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:32.470817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.470785    4925 patch_prober.go:29] interesting pod/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:32.470963 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.470836    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:32.620042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:32.620016    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"]
Oct 30 19:48:32.622294 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:48:32.622269    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68051b5f_7fa8_4a09_8bfe_54bbb43767f0.slice/crio-9ded0cbb3dffa979ad21a57106e092944aa2ad493d73111aee2cf567c54f8f53 WatchSource:0}: Error finding container 9ded0cbb3dffa979ad21a57106e092944aa2ad493d73111aee2cf567c54f8f53: Status 404 returned error can't find the container with id 9ded0cbb3dffa979ad21a57106e092944aa2ad493d73111aee2cf567c54f8f53
Oct 30 19:48:33.168534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:33.168444    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:33.203344 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:33.203310    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd" event={"ID":"68051b5f-7fa8-4a09-8bfe-54bbb43767f0","Type":"ContainerStarted","Data":"4d3a787599574a3d5023c28655cf16edde8774546f64f7d0e0db25bf8b2d7e9e"}
Oct 30 19:48:33.203487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:33.203374    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd" event={"ID":"68051b5f-7fa8-4a09-8bfe-54bbb43767f0","Type":"ContainerStarted","Data":"9ded0cbb3dffa979ad21a57106e092944aa2ad493d73111aee2cf567c54f8f53"}
Oct 30 19:48:33.222352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:33.222302    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd" podStartSLOduration=2.222288265 podStartE2EDuration="2.222288265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:33.219726127 +0000 UTC m=+2548.122739163" watchObservedRunningTime="2026-10-30 19:48:33.222288265 +0000 UTC m=+2548.125301303"
Oct 30 19:48:33.764194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:33.764108    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:34.037719 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.037580    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:34.209090 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.209052    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:34.213447 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.213406    4925 generic.go:356] "Generic (PLEG): container finished" podID="68051b5f-7fa8-4a09-8bfe-54bbb43767f0" containerID="4d3a787599574a3d5023c28655cf16edde8774546f64f7d0e0db25bf8b2d7e9e" exitCode=0
Oct 30 19:48:34.213616 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.213482    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd" event={"ID":"68051b5f-7fa8-4a09-8bfe-54bbb43767f0","Type":"ContainerDied","Data":"4d3a787599574a3d5023c28655cf16edde8774546f64f7d0e0db25bf8b2d7e9e"}
Oct 30 19:48:34.384497 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.384412    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:34.458185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.458153    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"]
Oct 30 19:48:34.469362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.469324    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-27b5d5-qsfgq"]
Oct 30 19:48:34.561716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.561235    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:34.837143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.837108    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:34.925449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.925418    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"]
Oct 30 19:48:34.933670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:34.933647    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2d5d53-qt8lq"]
Oct 30 19:48:35.007589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.007551    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:35.174325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.174287    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:35.191879 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.191841    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk" podUID="6bd9fde7-3646-4e83-a958-9aad45c10daf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:35.192026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.191850    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h" podUID="55bc197f-e609-4a15-a323-194cc2fbbf95" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:35.263099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.263055    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ba9701-eed9-402f-b28c-9ebe9b6e72ae" path="/var/lib/kubelet/pods/32ba9701-eed9-402f-b28c-9ebe9b6e72ae/volumes"
Oct 30 19:48:35.263746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.263724    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79867306-8d8e-44be-9f23-71ae7f9c5ad7" path="/var/lib/kubelet/pods/79867306-8d8e-44be-9f23-71ae7f9c5ad7/volumes"
Oct 30 19:48:35.360566 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.360542    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:35.446014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.445943    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-kube-api-access-74255\" (UniqueName: \"kubernetes.io/projected/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-kube-api-access-74255\") pod \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") "
Oct 30 19:48:35.446151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.446013    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-openshift-ca\") pod \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") "
Oct 30 19:48:35.446151 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.446121    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-xks-ca\") pod \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\" (UID: \"68051b5f-7fa8-4a09-8bfe-54bbb43767f0\") "
Oct 30 19:48:35.446413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.446388    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-openshift-ca" pod "68051b5f-7fa8-4a09-8bfe-54bbb43767f0" (UID: "68051b5f-7fa8-4a09-8bfe-54bbb43767f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:35.448240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.448214    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-xks-ca" pod "68051b5f-7fa8-4a09-8bfe-54bbb43767f0" (UID: "68051b5f-7fa8-4a09-8bfe-54bbb43767f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:35.448376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.448334    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-kube-api-access-74255" pod "68051b5f-7fa8-4a09-8bfe-54bbb43767f0" (UID: "68051b5f-7fa8-4a09-8bfe-54bbb43767f0"). InnerVolumeSpecName "kube-api-access-74255". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:35.525890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.525854    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:35.547365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.547336    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:35.547365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.547361    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74255\" (UniqueName: \"kubernetes.io/projected/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-kube-api-access-74255\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:35.547512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:35.547378    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68051b5f-7fa8-4a09-8bfe-54bbb43767f0-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.042311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.042268    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:36.127729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.127700    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:48:36.161939 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.161913    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:48:36.228371 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.228337    4925 generic.go:356] "Generic (PLEG): container finished" podID="55bc197f-e609-4a15-a323-194cc2fbbf95" containerID="ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f" exitCode=0
Oct 30 19:48:36.228550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.228380    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"
Oct 30 19:48:36.228550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.228419    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h" event={"ID":"55bc197f-e609-4a15-a323-194cc2fbbf95","Type":"ContainerDied","Data":"ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f"}
Oct 30 19:48:36.228550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.228497    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h" event={"ID":"55bc197f-e609-4a15-a323-194cc2fbbf95","Type":"ContainerDied","Data":"7d1fe3077b0f5485d372e6fe79e06c8b4242674f43233d0587ad2c7e6d62a73a"}
Oct 30 19:48:36.228550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.228526    4925 scope.go:122] "RemoveContainer" containerID="ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f"
Oct 30 19:48:36.229910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.229874    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"
Oct 30 19:48:36.229991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.229908    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd" event={"ID":"68051b5f-7fa8-4a09-8bfe-54bbb43767f0","Type":"ContainerDied","Data":"9ded0cbb3dffa979ad21a57106e092944aa2ad493d73111aee2cf567c54f8f53"}
Oct 30 19:48:36.229991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.229944    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ded0cbb3dffa979ad21a57106e092944aa2ad493d73111aee2cf567c54f8f53"
Oct 30 19:48:36.231349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.231328    4925 generic.go:356] "Generic (PLEG): container finished" podID="6bd9fde7-3646-4e83-a958-9aad45c10daf" containerID="60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78" exitCode=0
Oct 30 19:48:36.231473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.231376    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"
Oct 30 19:48:36.231473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.231412    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk" event={"ID":"6bd9fde7-3646-4e83-a958-9aad45c10daf","Type":"ContainerDied","Data":"60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78"}
Oct 30 19:48:36.231473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.231460    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk" event={"ID":"6bd9fde7-3646-4e83-a958-9aad45c10daf","Type":"ContainerDied","Data":"8c9bad36468e2d3e086873528637c9513de544e98e10d902358d82d24eef55fa"}
Oct 30 19:48:36.240262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.240244    4925 scope.go:122] "RemoveContainer" containerID="ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f"
Oct 30 19:48:36.240512 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:36.240497    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f\": container with ID starting with ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f not found: ID does not exist" containerID="ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f"
Oct 30 19:48:36.240561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.240518    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f"} err="failed to get container status \"ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f\": rpc error: code = NotFound desc = could not find container \"ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f\": container with ID starting with ea7ff4855f2f036a63c51731a14f4c8b3297c5d41e941cec7b52d2c552cd1d3f not found: ID does not exist"
Oct 30 19:48:36.240561 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.240532    4925 scope.go:122] "RemoveContainer" containerID="60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78"
Oct 30 19:48:36.249748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.249733    4925 scope.go:122] "RemoveContainer" containerID="60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78"
Oct 30 19:48:36.250009 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:36.249990    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78\": container with ID starting with 60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78 not found: ID does not exist" containerID="60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78"
Oct 30 19:48:36.250096 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.250017    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78"} err="failed to get container status \"60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78\": rpc error: code = NotFound desc = could not find container \"60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78\": container with ID starting with 60a99e7c985a10cea71db008a2d5fe80e4a88552eb1393789789bd1a3b859b78 not found: ID does not exist"
Oct 30 19:48:36.255389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.255368    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd9fde7-3646-4e83-a958-9aad45c10daf-kube-api-access-qrz8h\" (UniqueName: \"kubernetes.io/projected/6bd9fde7-3646-4e83-a958-9aad45c10daf-kube-api-access-qrz8h\") pod \"6bd9fde7-3646-4e83-a958-9aad45c10daf\" (UID: \"6bd9fde7-3646-4e83-a958-9aad45c10daf\") "
Oct 30 19:48:36.255584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.255570    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd9fde7-3646-4e83-a958-9aad45c10daf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd9fde7-3646-4e83-a958-9aad45c10daf-plugins-config-volume\") pod \"6bd9fde7-3646-4e83-a958-9aad45c10daf\" (UID: \"6bd9fde7-3646-4e83-a958-9aad45c10daf\") "
Oct 30 19:48:36.255653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.255604    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55bc197f-e609-4a15-a323-194cc2fbbf95-kube-api-access-phxvp\" (UniqueName: \"kubernetes.io/projected/55bc197f-e609-4a15-a323-194cc2fbbf95-kube-api-access-phxvp\") pod \"55bc197f-e609-4a15-a323-194cc2fbbf95\" (UID: \"55bc197f-e609-4a15-a323-194cc2fbbf95\") "
Oct 30 19:48:36.255653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.255638    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55bc197f-e609-4a15-a323-194cc2fbbf95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc197f-e609-4a15-a323-194cc2fbbf95-plugins-config-volume\") pod \"55bc197f-e609-4a15-a323-194cc2fbbf95\" (UID: \"55bc197f-e609-4a15-a323-194cc2fbbf95\") "
Oct 30 19:48:36.256017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.255993    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd9fde7-3646-4e83-a958-9aad45c10daf-plugins-config-volume" pod "6bd9fde7-3646-4e83-a958-9aad45c10daf" (UID: "6bd9fde7-3646-4e83-a958-9aad45c10daf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:36.256076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.255995    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55bc197f-e609-4a15-a323-194cc2fbbf95-plugins-config-volume" pod "55bc197f-e609-4a15-a323-194cc2fbbf95" (UID: "55bc197f-e609-4a15-a323-194cc2fbbf95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:36.257573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.257548    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd9fde7-3646-4e83-a958-9aad45c10daf-kube-api-access-qrz8h" pod "6bd9fde7-3646-4e83-a958-9aad45c10daf" (UID: "6bd9fde7-3646-4e83-a958-9aad45c10daf"). InnerVolumeSpecName "kube-api-access-qrz8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:36.257677 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.257609    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55bc197f-e609-4a15-a323-194cc2fbbf95-kube-api-access-phxvp" pod "55bc197f-e609-4a15-a323-194cc2fbbf95" (UID: "55bc197f-e609-4a15-a323-194cc2fbbf95"). InnerVolumeSpecName "kube-api-access-phxvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:36.357295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.357267    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrz8h\" (UniqueName: \"kubernetes.io/projected/6bd9fde7-3646-4e83-a958-9aad45c10daf-kube-api-access-qrz8h\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.357418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.357300    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bd9fde7-3646-4e83-a958-9aad45c10daf-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.357418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.357312    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxvp\" (UniqueName: \"kubernetes.io/projected/55bc197f-e609-4a15-a323-194cc2fbbf95-kube-api-access-phxvp\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.357418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.357321    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc197f-e609-4a15-a323-194cc2fbbf95-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.602952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.602925    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"]
Oct 30 19:48:36.606768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.606745    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b281d5-6b596cff6c-gth9h"]
Oct 30 19:48:36.616639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.616613    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"]
Oct 30 19:48:36.621231 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.621207    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b281d5-d8d4b4ff6-46rzk"]
Oct 30 19:48:36.812592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.812570    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:48:36.866559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.866007    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:36.963023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.962993    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-kube-api-access-nx696\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-kube-api-access-nx696\") pod \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") "
Oct 30 19:48:36.963023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963026    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istiod-ca-cert\") pod \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") "
Oct 30 19:48:36.963256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963059    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-socket\") pod \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") "
Oct 30 19:48:36.963256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963118    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-token\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-token\") pod \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") "
Oct 30 19:48:36.963256 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963201    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-data\") pod \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") "
Oct 30 19:48:36.963428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963294    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-certs\") pod \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") "
Oct 30 19:48:36.963428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963327    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-podinfo\") pod \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") "
Oct 30 19:48:36.963537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963446    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-envoy\") pod \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") "
Oct 30 19:48:36.963537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963480    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-data" pod "206b13a6-c1fc-42a0-8f34-01b580c0afbd" (UID: "206b13a6-c1fc-42a0-8f34-01b580c0afbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:36.963537 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963497    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-credential-socket\") pod \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\" (UID: \"206b13a6-c1fc-42a0-8f34-01b580c0afbd\") "
Oct 30 19:48:36.963769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963511    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istiod-ca-cert" pod "206b13a6-c1fc-42a0-8f34-01b580c0afbd" (UID: "206b13a6-c1fc-42a0-8f34-01b580c0afbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:36.963769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963520    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-socket" pod "206b13a6-c1fc-42a0-8f34-01b580c0afbd" (UID: "206b13a6-c1fc-42a0-8f34-01b580c0afbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:36.963769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963593    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-certs" pod "206b13a6-c1fc-42a0-8f34-01b580c0afbd" (UID: "206b13a6-c1fc-42a0-8f34-01b580c0afbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:36.963769 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.963747    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-credential-socket" pod "206b13a6-c1fc-42a0-8f34-01b580c0afbd" (UID: "206b13a6-c1fc-42a0-8f34-01b580c0afbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:36.964158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.964105    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.964158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.964125    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.964158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.964147    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.964343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.964163    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.964343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.964178    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:36.965658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.965631    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-token" pod "206b13a6-c1fc-42a0-8f34-01b580c0afbd" (UID: "206b13a6-c1fc-42a0-8f34-01b580c0afbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:36.965790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.965665    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-kube-api-access-nx696" pod "206b13a6-c1fc-42a0-8f34-01b580c0afbd" (UID: "206b13a6-c1fc-42a0-8f34-01b580c0afbd"). InnerVolumeSpecName "kube-api-access-nx696". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:36.965790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.965736    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-podinfo" pod "206b13a6-c1fc-42a0-8f34-01b580c0afbd" (UID: "206b13a6-c1fc-42a0-8f34-01b580c0afbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:36.965874 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:36.965815    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-envoy" pod "206b13a6-c1fc-42a0-8f34-01b580c0afbd" (UID: "206b13a6-c1fc-42a0-8f34-01b580c0afbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:37.065046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.064972    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:37.065046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.065000    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:37.065046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.065009    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/206b13a6-c1fc-42a0-8f34-01b580c0afbd-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:37.065046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.065021    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx696\" (UniqueName: \"kubernetes.io/projected/206b13a6-c1fc-42a0-8f34-01b580c0afbd-kube-api-access-nx696\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:37.241666 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.241631    4925 generic.go:356] "Generic (PLEG): container finished" podID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerID="205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8" exitCode=0
Oct 30 19:48:37.241876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.241681    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"
Oct 30 19:48:37.241876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.241738    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" event={"ID":"206b13a6-c1fc-42a0-8f34-01b580c0afbd","Type":"ContainerDied","Data":"205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8"}
Oct 30 19:48:37.241876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.241803    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k" event={"ID":"206b13a6-c1fc-42a0-8f34-01b580c0afbd","Type":"ContainerDied","Data":"4609f41b91ab3f942144f9dd039ff19317904f58ca02bfcacc3cae1b6ccbd4d3"}
Oct 30 19:48:37.241876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.241835    4925 scope.go:122] "RemoveContainer" containerID="205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8"
Oct 30 19:48:37.253498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.253482    4925 scope.go:122] "RemoveContainer" containerID="0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d"
Oct 30 19:48:37.263788 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.263658    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55bc197f-e609-4a15-a323-194cc2fbbf95" path="/var/lib/kubelet/pods/55bc197f-e609-4a15-a323-194cc2fbbf95/volumes"
Oct 30 19:48:37.263883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.263804    4925 scope.go:122] "RemoveContainer" containerID="205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8"
Oct 30 19:48:37.264141 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:37.264117    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8\": container with ID starting with 205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8 not found: ID does not exist" containerID="205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8"
Oct 30 19:48:37.264197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.264155    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8"} err="failed to get container status \"205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8\": rpc error: code = NotFound desc = could not find container \"205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8\": container with ID starting with 205933e7b5f7f830008c4036140be947d96f7b113a0c1952e304701fdd0ceab8 not found: ID does not exist"
Oct 30 19:48:37.264197 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.264178    4925 scope.go:122] "RemoveContainer" containerID="0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d"
Oct 30 19:48:37.264292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.264199    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd9fde7-3646-4e83-a958-9aad45c10daf" path="/var/lib/kubelet/pods/6bd9fde7-3646-4e83-a958-9aad45c10daf/volumes"
Oct 30 19:48:37.264499 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:37.264474    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d\": container with ID starting with 0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d not found: ID does not exist" containerID="0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d"
Oct 30 19:48:37.264609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.264503    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d"} err="failed to get container status \"0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d\": rpc error: code = NotFound desc = could not find container \"0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d\": container with ID starting with 0dea77124237137026587099909844787d04c9715d81c40af44e5ba76ad1821d not found: ID does not exist"
Oct 30 19:48:37.264609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.264589    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"]
Oct 30 19:48:37.271831 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.271810    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-be6b74-openshift-default-954dcdd97-w4h6k"]
Oct 30 19:48:37.575082 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:37.575025    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:38.090644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:38.090605    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:38.280202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:38.280172    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:38.455264 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:38.455230    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:38.640033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:38.639999    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:38.840046 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:38.839962    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:39.278778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:39.278722    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" path="/var/lib/kubelet/pods/206b13a6-c1fc-42a0-8f34-01b580c0afbd/volumes"
Oct 30 19:48:39.282397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:39.282340    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:40.393428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:40.393394    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:41.149907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:41.149864    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:41.865306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:41.865266    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:42.303061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:42.303028    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:42.471632 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:42.471601    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:42.667377 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:42.667345    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:42.706815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:42.706784    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"]
Oct 30 19:48:42.709582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:42.709556    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ddb6ea-pcsjd"]
Oct 30 19:48:42.836602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:42.836564    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:43.009431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.009341    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:43.262376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.262299    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68051b5f-7fa8-4a09-8bfe-54bbb43767f0" path="/var/lib/kubelet/pods/68051b5f-7fa8-4a09-8bfe-54bbb43767f0/volumes"
Oct 30 19:48:43.307045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.307004    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:43.489470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.489436    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:43.652311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.652270    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:43.798542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.798496    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"]
Oct 30 19:48:43.798833 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.798806    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy" containerID="cri-o://829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3" gracePeriod=30
Oct 30 19:48:43.907771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.904087    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:43.918519 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.918485    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"]
Oct 30 19:48:43.918818 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:43.918792    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy" containerID="cri-o://b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853" gracePeriod=30
Oct 30 19:48:44.032949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:44.032909    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"]
Oct 30 19:48:44.037359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:44.037331    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547898e1-zq6kd"]
Oct 30 19:48:44.155710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:44.152410    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:44.697750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:44.697680    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:45.261660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:45.261628    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6f42e1-b321-4c90-89a7-7a9a020189e9" path="/var/lib/kubelet/pods/cb6f42e1-b321-4c90-89a7-7a9a020189e9/volumes"
Oct 30 19:48:45.325206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:45.325167    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:46.081612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:46.081498    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:46.253493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:46.253461    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:46.455755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:46.455722    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"]
Oct 30 19:48:46.456590 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:46.456545    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy" containerID="cri-o://2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47" gracePeriod=30
Oct 30 19:48:46.557664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:46.557611    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:46.792054 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:46.791972    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:47.071673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:47.071584    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:47.735068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:47.735032    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:48.431146 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:48.431110    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:48.601350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:48.601316    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:49.077044 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.076987    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:48:49.141729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.141662    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-data\") pod \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") "
Oct 30 19:48:49.141729 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.141722    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-podinfo\") pod \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") "
Oct 30 19:48:49.141984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.141789    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-kube-api-access-fdrjs\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-kube-api-access-fdrjs\") pod \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") "
Oct 30 19:48:49.141984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.141846    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-credential-socket\") pod \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") "
Oct 30 19:48:49.141984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.141878    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-certs\") pod \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") "
Oct 30 19:48:49.141984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.141910    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-envoy\") pod \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") "
Oct 30 19:48:49.141984 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.141967    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istiod-ca-cert\") pod \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") "
Oct 30 19:48:49.142271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.141999    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-socket\") pod \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") "
Oct 30 19:48:49.142271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.142121    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-token\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-token\") pod \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\" (UID: \"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915\") "
Oct 30 19:48:49.142271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.141965    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-data" pod "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" (UID: "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.142271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.142132    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-credential-socket" pod "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" (UID: "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.142648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.142617    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.142648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.142645    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.143352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.143013    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istiod-ca-cert" pod "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" (UID: "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:49.143352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.143084    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-socket" pod "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" (UID: "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.143352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.143326    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-certs" pod "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" (UID: "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.145234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.145198    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-envoy" pod "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" (UID: "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.145391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.145262    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-token" pod "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" (UID: "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:49.145675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.145654    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-kube-api-access-fdrjs" pod "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" (UID: "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915"). InnerVolumeSpecName "kube-api-access-fdrjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:49.145675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.145659    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-podinfo" pod "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" (UID: "6aeb17ec-a2bf-4a76-8d21-7edc62e2c915"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:49.192622 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.192600    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:48:49.243652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.243615    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-istio-token\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-istio-token\") pod \"8411f4a9-165d-47a5-a255-e6d07376731e\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") "
Oct 30 19:48:49.243839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.243662    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8411f4a9-165d-47a5-a255-e6d07376731e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8411f4a9-165d-47a5-a255-e6d07376731e-istiod-ca-cert\") pod \"8411f4a9-165d-47a5-a255-e6d07376731e\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") "
Oct 30 19:48:49.243839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.243714    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-envoy\") pod \"8411f4a9-165d-47a5-a255-e6d07376731e\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") "
Oct 30 19:48:49.243839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.243786    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-kube-api-access-gn8sz\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-kube-api-access-gn8sz\") pod \"8411f4a9-165d-47a5-a255-e6d07376731e\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") "
Oct 30 19:48:49.244093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.243868    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8411f4a9-165d-47a5-a255-e6d07376731e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8411f4a9-165d-47a5-a255-e6d07376731e-istio-podinfo\") pod \"8411f4a9-165d-47a5-a255-e6d07376731e\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") "
Oct 30 19:48:49.244093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.243912    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-credential-socket\") pod \"8411f4a9-165d-47a5-a255-e6d07376731e\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") "
Oct 30 19:48:49.244093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244001    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-data\") pod \"8411f4a9-165d-47a5-a255-e6d07376731e\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") "
Oct 30 19:48:49.244733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244090    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-socket\") pod \"8411f4a9-165d-47a5-a255-e6d07376731e\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") "
Oct 30 19:48:49.244733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244128    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-certs\") pod \"8411f4a9-165d-47a5-a255-e6d07376731e\" (UID: \"8411f4a9-165d-47a5-a255-e6d07376731e\") "
Oct 30 19:48:49.244733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244302    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8411f4a9-165d-47a5-a255-e6d07376731e-istiod-ca-cert" pod "8411f4a9-165d-47a5-a255-e6d07376731e" (UID: "8411f4a9-165d-47a5-a255-e6d07376731e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:49.244733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244449    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-socket" pod "8411f4a9-165d-47a5-a255-e6d07376731e" (UID: "8411f4a9-165d-47a5-a255-e6d07376731e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.244733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244517    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-data" pod "8411f4a9-165d-47a5-a255-e6d07376731e" (UID: "8411f4a9-165d-47a5-a255-e6d07376731e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.244733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244597    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-certs" pod "8411f4a9-165d-47a5-a255-e6d07376731e" (UID: "8411f4a9-165d-47a5-a255-e6d07376731e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.244733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244706    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-credential-socket" pod "8411f4a9-165d-47a5-a255-e6d07376731e" (UID: "8411f4a9-165d-47a5-a255-e6d07376731e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244798    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244817    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244832    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244845    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244857    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244871    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244883    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244896    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8411f4a9-165d-47a5-a255-e6d07376731e-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244910    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrjs\" (UniqueName: \"kubernetes.io/projected/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-kube-api-access-fdrjs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244923    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.245216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.244935    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.246847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.246826    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-envoy" pod "8411f4a9-165d-47a5-a255-e6d07376731e" (UID: "8411f4a9-165d-47a5-a255-e6d07376731e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:49.246847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.246836    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-istio-token" pod "8411f4a9-165d-47a5-a255-e6d07376731e" (UID: "8411f4a9-165d-47a5-a255-e6d07376731e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:49.247249 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.247230    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-kube-api-access-gn8sz" pod "8411f4a9-165d-47a5-a255-e6d07376731e" (UID: "8411f4a9-165d-47a5-a255-e6d07376731e"). InnerVolumeSpecName "kube-api-access-gn8sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:49.247349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.247313    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8411f4a9-165d-47a5-a255-e6d07376731e-istio-podinfo" pod "8411f4a9-165d-47a5-a255-e6d07376731e" (UID: "8411f4a9-165d-47a5-a255-e6d07376731e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:49.345609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.345538    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8411f4a9-165d-47a5-a255-e6d07376731e-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.345609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.345570    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.345609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.345581    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.345609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.345590    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8411f4a9-165d-47a5-a255-e6d07376731e-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.345609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.345599    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn8sz\" (UniqueName: \"kubernetes.io/projected/8411f4a9-165d-47a5-a255-e6d07376731e-kube-api-access-gn8sz\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:49.349102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.349070    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:49.367954 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.367922    4925 generic.go:356] "Generic (PLEG): container finished" podID="8411f4a9-165d-47a5-a255-e6d07376731e" containerID="b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853" exitCode=0
Oct 30 19:48:49.368121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.367977    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"
Oct 30 19:48:49.368121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.368006    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" event={"ID":"8411f4a9-165d-47a5-a255-e6d07376731e","Type":"ContainerDied","Data":"b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853"}
Oct 30 19:48:49.368121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.368080    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj" event={"ID":"8411f4a9-165d-47a5-a255-e6d07376731e","Type":"ContainerDied","Data":"9880e509c4052ae9ea646cf77dd6154ac7702183a911bb51365b8ea6cb6ae12e"}
Oct 30 19:48:49.368121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.368103    4925 scope.go:122] "RemoveContainer" containerID="b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853"
Oct 30 19:48:49.369664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.369641    4925 generic.go:356] "Generic (PLEG): container finished" podID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerID="829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3" exitCode=0
Oct 30 19:48:49.369803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.369745    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"
Oct 30 19:48:49.369880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.369822    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" event={"ID":"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915","Type":"ContainerDied","Data":"829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3"}
Oct 30 19:48:49.369880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.369869    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9" event={"ID":"6aeb17ec-a2bf-4a76-8d21-7edc62e2c915","Type":"ContainerDied","Data":"f8369943fd75e2ded6ee87a45376adf83db36879cce9b1bd802d32e02ed3d5a8"}
Oct 30 19:48:49.378657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.378642    4925 scope.go:122] "RemoveContainer" containerID="105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa"
Oct 30 19:48:49.388065 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.388047    4925 scope.go:122] "RemoveContainer" containerID="b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853"
Oct 30 19:48:49.388309 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:49.388291    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853\": container with ID starting with b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853 not found: ID does not exist" containerID="b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853"
Oct 30 19:48:49.388434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.388318    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853"} err="failed to get container status \"b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853\": rpc error: code = NotFound desc = could not find container \"b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853\": container with ID starting with b47b9ffe2516719e4df3add4258be9f7f0c483d5a904a3488e1f456fa2b29853 not found: ID does not exist"
Oct 30 19:48:49.388434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.388333    4925 scope.go:122] "RemoveContainer" containerID="105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa"
Oct 30 19:48:49.388576 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:49.388559    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa\": container with ID starting with 105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa not found: ID does not exist" containerID="105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa"
Oct 30 19:48:49.388614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.388580    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa"} err="failed to get container status \"105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa\": rpc error: code = NotFound desc = could not find container \"105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa\": container with ID starting with 105e1702c4d7febb1deb6e03322f5c2d820b566dc933db417f74be30861585fa not found: ID does not exist"
Oct 30 19:48:49.388614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.388594    4925 scope.go:122] "RemoveContainer" containerID="829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3"
Oct 30 19:48:49.393203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.393184    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"]
Oct 30 19:48:49.398534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.398515    4925 scope.go:122] "RemoveContainer" containerID="829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3"
Oct 30 19:48:49.398868 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:49.398843    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3\": container with ID starting with 829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3 not found: ID does not exist" containerID="829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3"
Oct 30 19:48:49.398982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.398875    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3"} err="failed to get container status \"829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3\": rpc error: code = NotFound desc = could not find container \"829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3\": container with ID starting with 829f0e6c3ab3fb441a68a6be61fcdc987b3820b1243c24d9fd1f3c5e4bb65fd3 not found: ID does not exist"
Oct 30 19:48:49.399856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.399838    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-27b5d5-openshift-default-66b797556c-d9gs9"]
Oct 30 19:48:49.410653 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.410632    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"]
Oct 30 19:48:49.415255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:49.415236    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2d5d53-openshift-default-6dd7664849-jdlkj"]
Oct 30 19:48:50.456481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:50.456445    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:51.066016 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.065984    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:51.261790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.261763    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" path="/var/lib/kubelet/pods/6aeb17ec-a2bf-4a76-8d21-7edc62e2c915/volumes"
Oct 30 19:48:51.262190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.262176    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" path="/var/lib/kubelet/pods/8411f4a9-165d-47a5-a255-e6d07376731e/volumes"
Oct 30 19:48:51.759297 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.759273    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:48:51.885275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885242    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istiod-ca-cert\") pod \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") "
Oct 30 19:48:51.885455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885285    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-podinfo\") pod \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") "
Oct 30 19:48:51.885455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885315    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-socket\") pod \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") "
Oct 30 19:48:51.885455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885365    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-envoy\") pod \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") "
Oct 30 19:48:51.885455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885417    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-credential-socket\") pod \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") "
Oct 30 19:48:51.885641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885465    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-token\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-token\") pod \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") "
Oct 30 19:48:51.885641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885538    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-data\") pod \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") "
Oct 30 19:48:51.885641 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885578    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-certs\") pod \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") "
Oct 30 19:48:51.885837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885662    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-socket" pod "b4afab18-0b83-4db6-a92e-3ad1c5a50da9" (UID: "b4afab18-0b83-4db6-a92e-3ad1c5a50da9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:51.885837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885725    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istiod-ca-cert" pod "b4afab18-0b83-4db6-a92e-3ad1c5a50da9" (UID: "b4afab18-0b83-4db6-a92e-3ad1c5a50da9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:51.885837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885737    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-kube-api-access-g9fnk\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-kube-api-access-g9fnk\") pod \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\" (UID: \"b4afab18-0b83-4db6-a92e-3ad1c5a50da9\") "
Oct 30 19:48:51.886017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885842    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-data" pod "b4afab18-0b83-4db6-a92e-3ad1c5a50da9" (UID: "b4afab18-0b83-4db6-a92e-3ad1c5a50da9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:51.886017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.885984    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-credential-socket" pod "b4afab18-0b83-4db6-a92e-3ad1c5a50da9" (UID: "b4afab18-0b83-4db6-a92e-3ad1c5a50da9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:51.886141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.886017    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-certs" pod "b4afab18-0b83-4db6-a92e-3ad1c5a50da9" (UID: "b4afab18-0b83-4db6-a92e-3ad1c5a50da9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:51.886251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.886218    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.886502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.886255    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.886502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.886268    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.886502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.886280    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.886502 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.886293    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.887959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.887934    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-podinfo" pod "b4afab18-0b83-4db6-a92e-3ad1c5a50da9" (UID: "b4afab18-0b83-4db6-a92e-3ad1c5a50da9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:51.888071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.887992    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-token" pod "b4afab18-0b83-4db6-a92e-3ad1c5a50da9" (UID: "b4afab18-0b83-4db6-a92e-3ad1c5a50da9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:51.888406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.888377    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-envoy" pod "b4afab18-0b83-4db6-a92e-3ad1c5a50da9" (UID: "b4afab18-0b83-4db6-a92e-3ad1c5a50da9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:51.888406 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.888384    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-kube-api-access-g9fnk" pod "b4afab18-0b83-4db6-a92e-3ad1c5a50da9" (UID: "b4afab18-0b83-4db6-a92e-3ad1c5a50da9"). InnerVolumeSpecName "kube-api-access-g9fnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:51.987313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.987233    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9fnk\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-kube-api-access-g9fnk\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.987313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.987264    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.987313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.987274    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:51.987313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:51.987283    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4afab18-0b83-4db6-a92e-3ad1c5a50da9-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:52.049785 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.049750    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"]
Oct 30 19:48:52.060347 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.058570    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"]
Oct 30 19:48:52.061933 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.061901    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:52.067175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.067136    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"]
Oct 30 19:48:52.082952 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.082922    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"]
Oct 30 19:48:52.139953 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:52.139921    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:48:52.238767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.235567    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:52.395900 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.395871    4925 generic.go:356] "Generic (PLEG): container finished" podID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerID="2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47" exitCode=0
Oct 30 19:48:52.396055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.395907    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" event={"ID":"b4afab18-0b83-4db6-a92e-3ad1c5a50da9","Type":"ContainerDied","Data":"2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47"}
Oct 30 19:48:52.396055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.395940    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"
Oct 30 19:48:52.396055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.395973    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n" event={"ID":"b4afab18-0b83-4db6-a92e-3ad1c5a50da9","Type":"ContainerDied","Data":"af7d223ff90bd27e0d013b6ea3364dc511204e8560c4940c5cbd5af0b7d16e1c"}
Oct 30 19:48:52.396055 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.396003    4925 scope.go:122] "RemoveContainer" containerID="2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47"
Oct 30 19:48:52.405006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.404965    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:52.407884 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.407868    4925 scope.go:122] "RemoveContainer" containerID="e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd"
Oct 30 19:48:52.417815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.417766    4925 scope.go:122] "RemoveContainer" containerID="2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47"
Oct 30 19:48:52.418040 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:52.418022    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47\": container with ID starting with 2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47 not found: ID does not exist" containerID="2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47"
Oct 30 19:48:52.418098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.418049    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47"} err="failed to get container status \"2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47\": rpc error: code = NotFound desc = could not find container \"2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47\": container with ID starting with 2fb27b409745c8ae57dad7d2f62678f8f091c17fda7532ec2c77b958a8984d47 not found: ID does not exist"
Oct 30 19:48:52.418098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.418066    4925 scope.go:122] "RemoveContainer" containerID="e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd"
Oct 30 19:48:52.418292 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:52.418275    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd\": container with ID starting with e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd not found: ID does not exist" containerID="e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd"
Oct 30 19:48:52.418336 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.418297    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd"} err="failed to get container status \"e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd\": rpc error: code = NotFound desc = could not find container \"e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd\": container with ID starting with e839ad6e0fbe5e37b7518c1a6be0d12f3f4f9578a543d5c152a39c520fd011bd not found: ID does not exist"
Oct 30 19:48:52.424777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.424748    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"]
Oct 30 19:48:52.428337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.428311    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8f3ff4-openshift-default-74bf967ff9-9rf8n"]
Oct 30 19:48:52.577876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.577783    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:52.983745 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:52.983704    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:53.261762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:53.261650    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" path="/var/lib/kubelet/pods/b4afab18-0b83-4db6-a92e-3ad1c5a50da9/volumes"
Oct 30 19:48:53.496262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:53.496224    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:53.667623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:53.667587    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:53.808484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:53.808449    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"]
Oct 30 19:48:53.808755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:53.808721    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" containerID="cri-o://2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f" gracePeriod=30
Oct 30 19:48:53.813801 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:53.813770    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"]
Oct 30 19:48:53.814031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:53.814011    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy" containerID="cri-o://3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621" gracePeriod=30
Oct 30 19:48:53.912484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:53.912416    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:54.090294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.090213    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"]
Oct 30 19:48:54.105202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.105167    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee22e36a-v8djm"]
Oct 30 19:48:54.402358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402323    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"]
Oct 30 19:48:54.402837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402814    4925 cpu_manager.go:414] "containerMap: removing container" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy"
Oct 30 19:48:54.402837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402837    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402858    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402863    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402874    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402882    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402896    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6bd9fde7-3646-4e83-a958-9aad45c10daf" containerName="payload-processing"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402902    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd9fde7-3646-4e83-a958-9aad45c10daf" containerName="payload-processing"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402908    4925 cpu_manager.go:414] "containerMap: removing container" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402912    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402932    4925 cpu_manager.go:414] "containerMap: removing container" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402938    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402954    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402962    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402972    4925 cpu_manager.go:414] "containerMap: removing container" podUID="68051b5f-7fa8-4a09-8bfe-54bbb43767f0" containerName="revoke-keys"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402978    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68051b5f-7fa8-4a09-8bfe-54bbb43767f0" containerName="revoke-keys"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402991    4925 cpu_manager.go:414] "containerMap: removing container" podUID="55bc197f-e609-4a15-a323-194cc2fbbf95" containerName="payload-pre-processing"
Oct 30 19:48:54.403048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.402996    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55bc197f-e609-4a15-a323-194cc2fbbf95" containerName="payload-pre-processing"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403071    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403080    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8313268-9549-42a2-942b-aac4e43c75a3" containerName="istio-proxy"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403088    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403093    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403099    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd9fde7-3646-4e83-a958-9aad45c10daf" containerName="payload-processing"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403104    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aeb17ec-a2bf-4a76-8d21-7edc62e2c915" containerName="istio-proxy"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403109    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="206b13a6-c1fc-42a0-8f34-01b580c0afbd" containerName="istio-proxy"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403118    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="68051b5f-7fa8-4a09-8bfe-54bbb43767f0" containerName="revoke-keys"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403128    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy"
Oct 30 19:48:54.403965 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.403134    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="55bc197f-e609-4a15-a323-194cc2fbbf95" containerName="payload-pre-processing"
Oct 30 19:48:54.408114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.408096    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.411584 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.411559    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:54.411854 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.411820    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:54.415372 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.415352    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"]
Oct 30 19:48:54.505318 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.505283    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:54.505503 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.505336    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:54.516865 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.516834    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99659\" (UniqueName: \"kubernetes.io/projected/b4f0210a-e252-4aa6-8875-68a82b9a51d0-kube-api-access-99659\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.517032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.516896    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.517032 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.517011    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.617628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.617581    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99659\" (UniqueName: \"kubernetes.io/projected/b4f0210a-e252-4aa6-8875-68a82b9a51d0-kube-api-access-99659\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.617841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.617679    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.617841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.617763    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.618462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.618436    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.620374 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.620348    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.629596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.629575    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99659\" (UniqueName: \"kubernetes.io/projected/b4f0210a-e252-4aa6-8875-68a82b9a51d0-kube-api-access-99659\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.725219 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.725124    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:54.912319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:54.912275    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:55.011368 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:55.011143    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"]
Oct 30 19:48:55.014245 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:48:55.014202    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f0210a_e252_4aa6_8875_68a82b9a51d0.slice/crio-bde34bb5da62da245054ee1ad6aad14a3671d59aa159a655d3de269116edc362 WatchSource:0}: Error finding container bde34bb5da62da245054ee1ad6aad14a3671d59aa159a655d3de269116edc362: Status 404 returned error can't find the container with id bde34bb5da62da245054ee1ad6aad14a3671d59aa159a655d3de269116edc362
Oct 30 19:48:55.017916 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:55.017888    4925 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:55.018029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:55.017939    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:55.262040 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:55.261969    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ae576a-0cb2-4643-a6e5-bd6460de3a26" path="/var/lib/kubelet/pods/c0ae576a-0cb2-4643-a6e5-bd6460de3a26/volumes"
Oct 30 19:48:55.422714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:55.422647    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl" event={"ID":"b4f0210a-e252-4aa6-8875-68a82b9a51d0","Type":"ContainerStarted","Data":"8d6c26b9fd968c9cbc0440186b3236e702aee0cea8a07912ca3f61e5472cf148"}
Oct 30 19:48:55.422911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:55.422727    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl" event={"ID":"b4f0210a-e252-4aa6-8875-68a82b9a51d0","Type":"ContainerStarted","Data":"bde34bb5da62da245054ee1ad6aad14a3671d59aa159a655d3de269116edc362"}
Oct 30 19:48:55.440980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:55.440930    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl" podStartSLOduration=1.440917142 podStartE2EDuration="1.440917142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:55.439448294 +0000 UTC m=+2570.342461331" watchObservedRunningTime="2026-10-30 19:48:55.440917142 +0000 UTC m=+2570.343930208"
Oct 30 19:48:55.570956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:55.570856    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:56.272574 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:56.272488    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:56.432329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:56.432295    4925 generic.go:356] "Generic (PLEG): container finished" podID="b4f0210a-e252-4aa6-8875-68a82b9a51d0" containerID="8d6c26b9fd968c9cbc0440186b3236e702aee0cea8a07912ca3f61e5472cf148" exitCode=0
Oct 30 19:48:56.432527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:56.432377    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl" event={"ID":"b4f0210a-e252-4aa6-8875-68a82b9a51d0","Type":"ContainerDied","Data":"8d6c26b9fd968c9cbc0440186b3236e702aee0cea8a07912ca3f61e5472cf148"}
Oct 30 19:48:56.438133 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:56.438105    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:57.049384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.049343    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:57.252311 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.252276    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"]
Oct 30 19:48:57.366489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.366459    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:57.575031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.574999    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:57.620304 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.620279    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:57.752674 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.752643    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f0210a-e252-4aa6-8875-68a82b9a51d0-kube-api-access-99659\" (UniqueName: \"kubernetes.io/projected/b4f0210a-e252-4aa6-8875-68a82b9a51d0-kube-api-access-99659\") pod \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") "
Oct 30 19:48:57.752889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.752708    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-xks-ca\") pod \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") "
Oct 30 19:48:57.752889 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.752762    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-openshift-ca\") pod \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\" (UID: \"b4f0210a-e252-4aa6-8875-68a82b9a51d0\") "
Oct 30 19:48:57.753216 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.753185    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-openshift-ca" pod "b4f0210a-e252-4aa6-8875-68a82b9a51d0" (UID: "b4f0210a-e252-4aa6-8875-68a82b9a51d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:57.755125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.755095    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f0210a-e252-4aa6-8875-68a82b9a51d0-kube-api-access-99659" pod "b4f0210a-e252-4aa6-8875-68a82b9a51d0" (UID: "b4f0210a-e252-4aa6-8875-68a82b9a51d0"). InnerVolumeSpecName "kube-api-access-99659". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:57.755125 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.755112    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-xks-ca" pod "b4f0210a-e252-4aa6-8875-68a82b9a51d0" (UID: "b4f0210a-e252-4aa6-8875-68a82b9a51d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:57.815873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.815837    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:57.854899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.854820    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99659\" (UniqueName: \"kubernetes.io/projected/b4f0210a-e252-4aa6-8875-68a82b9a51d0-kube-api-access-99659\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:57.854899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.854847    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:57.854899 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:57.854861    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0210a-e252-4aa6-8875-68a82b9a51d0-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:58.453080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:58.452972    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"
Oct 30 19:48:58.453080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:58.452997    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl" event={"ID":"b4f0210a-e252-4aa6-8875-68a82b9a51d0","Type":"ContainerDied","Data":"bde34bb5da62da245054ee1ad6aad14a3671d59aa159a655d3de269116edc362"}
Oct 30 19:48:58.453080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:58.453053    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bde34bb5da62da245054ee1ad6aad14a3671d59aa159a655d3de269116edc362"
Oct 30 19:48:58.471765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:58.471729    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:58.637150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:58.637111    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:59.075431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.075344    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:48:59.113321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.113298    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:48:59.168623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.168585    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-kube-api-access-t9mvs\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-kube-api-access-t9mvs\") pod \"088a7b18-ab88-4099-a815-99c6f57835ed\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") "
Oct 30 19:48:59.168848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.168663    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-socket\") pod \"088a7b18-ab88-4099-a815-99c6f57835ed\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") "
Oct 30 19:48:59.168848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.168708    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-credential-socket\") pod \"088a7b18-ab88-4099-a815-99c6f57835ed\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") "
Oct 30 19:48:59.168848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.168726    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-istio-token\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-istio-token\") pod \"088a7b18-ab88-4099-a815-99c6f57835ed\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") "
Oct 30 19:48:59.168848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.168765    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/088a7b18-ab88-4099-a815-99c6f57835ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088a7b18-ab88-4099-a815-99c6f57835ed-istio-podinfo\") pod \"088a7b18-ab88-4099-a815-99c6f57835ed\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") "
Oct 30 19:48:59.168848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.168787    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-certs\") pod \"088a7b18-ab88-4099-a815-99c6f57835ed\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") "
Oct 30 19:48:59.168848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.168812    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088a7b18-ab88-4099-a815-99c6f57835ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088a7b18-ab88-4099-a815-99c6f57835ed-istiod-ca-cert\") pod \"088a7b18-ab88-4099-a815-99c6f57835ed\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") "
Oct 30 19:48:59.169213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.168875    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-data\") pod \"088a7b18-ab88-4099-a815-99c6f57835ed\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") "
Oct 30 19:48:59.169213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.168954    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-envoy\") pod \"088a7b18-ab88-4099-a815-99c6f57835ed\" (UID: \"088a7b18-ab88-4099-a815-99c6f57835ed\") "
Oct 30 19:48:59.169739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.169637    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-socket" pod "088a7b18-ab88-4099-a815-99c6f57835ed" (UID: "088a7b18-ab88-4099-a815-99c6f57835ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.169894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.169855    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-data" pod "088a7b18-ab88-4099-a815-99c6f57835ed" (UID: "088a7b18-ab88-4099-a815-99c6f57835ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.169960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.169924    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-certs" pod "088a7b18-ab88-4099-a815-99c6f57835ed" (UID: "088a7b18-ab88-4099-a815-99c6f57835ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.170033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.169953    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088a7b18-ab88-4099-a815-99c6f57835ed-istiod-ca-cert" pod "088a7b18-ab88-4099-a815-99c6f57835ed" (UID: "088a7b18-ab88-4099-a815-99c6f57835ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:59.170175 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.170154    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-credential-socket" pod "088a7b18-ab88-4099-a815-99c6f57835ed" (UID: "088a7b18-ab88-4099-a815-99c6f57835ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.172312 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.172169    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/088a7b18-ab88-4099-a815-99c6f57835ed-istio-podinfo" pod "088a7b18-ab88-4099-a815-99c6f57835ed" (UID: "088a7b18-ab88-4099-a815-99c6f57835ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:59.172816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.172777    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-envoy" pod "088a7b18-ab88-4099-a815-99c6f57835ed" (UID: "088a7b18-ab88-4099-a815-99c6f57835ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.172921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.172783    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-istio-token" pod "088a7b18-ab88-4099-a815-99c6f57835ed" (UID: "088a7b18-ab88-4099-a815-99c6f57835ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:59.172921 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.172837    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-kube-api-access-t9mvs" pod "088a7b18-ab88-4099-a815-99c6f57835ed" (UID: "088a7b18-ab88-4099-a815-99c6f57835ed"). InnerVolumeSpecName "kube-api-access-t9mvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:59.270101 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270071    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d846076-2d4e-42fe-8909-b6fde688f282-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d846076-2d4e-42fe-8909-b6fde688f282-istio-podinfo\") pod \"5d846076-2d4e-42fe-8909-b6fde688f282\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") "
Oct 30 19:48:59.270280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270108    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-data\") pod \"5d846076-2d4e-42fe-8909-b6fde688f282\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") "
Oct 30 19:48:59.270280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270167    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-istio-token\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-istio-token\") pod \"5d846076-2d4e-42fe-8909-b6fde688f282\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") "
Oct 30 19:48:59.270280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270207    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-credential-socket\") pod \"5d846076-2d4e-42fe-8909-b6fde688f282\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") "
Oct 30 19:48:59.270280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270236    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-certs\") pod \"5d846076-2d4e-42fe-8909-b6fde688f282\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") "
Oct 30 19:48:59.270509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270294    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-socket\") pod \"5d846076-2d4e-42fe-8909-b6fde688f282\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") "
Oct 30 19:48:59.270509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270342    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-envoy\") pod \"5d846076-2d4e-42fe-8909-b6fde688f282\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") "
Oct 30 19:48:59.270509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270375    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-kube-api-access-lzjwg\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-kube-api-access-lzjwg\") pod \"5d846076-2d4e-42fe-8909-b6fde688f282\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") "
Oct 30 19:48:59.270509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270439    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d846076-2d4e-42fe-8909-b6fde688f282-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d846076-2d4e-42fe-8909-b6fde688f282-istiod-ca-cert\") pod \"5d846076-2d4e-42fe-8909-b6fde688f282\" (UID: \"5d846076-2d4e-42fe-8909-b6fde688f282\") "
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270767    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-certs" pod "5d846076-2d4e-42fe-8909-b6fde688f282" (UID: "5d846076-2d4e-42fe-8909-b6fde688f282"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270869    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9mvs\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-kube-api-access-t9mvs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270893    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270907    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270922    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088a7b18-ab88-4099-a815-99c6f57835ed-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270936    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088a7b18-ab88-4099-a815-99c6f57835ed-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270949    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270963    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088a7b18-ab88-4099-a815-99c6f57835ed-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270977    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270992    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088a7b18-ab88-4099-a815-99c6f57835ed-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.270991    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-socket" pod "5d846076-2d4e-42fe-8909-b6fde688f282" (UID: "5d846076-2d4e-42fe-8909-b6fde688f282"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.271064    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-credential-socket" pod "5d846076-2d4e-42fe-8909-b6fde688f282" (UID: "5d846076-2d4e-42fe-8909-b6fde688f282"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.271244 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.271129    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-data" pod "5d846076-2d4e-42fe-8909-b6fde688f282" (UID: "5d846076-2d4e-42fe-8909-b6fde688f282"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.272352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.271293    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d846076-2d4e-42fe-8909-b6fde688f282-istiod-ca-cert" pod "5d846076-2d4e-42fe-8909-b6fde688f282" (UID: "5d846076-2d4e-42fe-8909-b6fde688f282"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:59.273780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.273446    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-istio-token" pod "5d846076-2d4e-42fe-8909-b6fde688f282" (UID: "5d846076-2d4e-42fe-8909-b6fde688f282"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:59.273987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.273968    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-envoy" pod "5d846076-2d4e-42fe-8909-b6fde688f282" (UID: "5d846076-2d4e-42fe-8909-b6fde688f282"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:59.274308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.274283    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d846076-2d4e-42fe-8909-b6fde688f282-istio-podinfo" pod "5d846076-2d4e-42fe-8909-b6fde688f282" (UID: "5d846076-2d4e-42fe-8909-b6fde688f282"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:59.274994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.274965    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-kube-api-access-lzjwg" pod "5d846076-2d4e-42fe-8909-b6fde688f282" (UID: "5d846076-2d4e-42fe-8909-b6fde688f282"). InnerVolumeSpecName "kube-api-access-lzjwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:59.281056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.281031    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:59.372220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.372142    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d846076-2d4e-42fe-8909-b6fde688f282-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.372220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.372165    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.372220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.372178    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.372220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.372191    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.372220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.372200    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.372220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.372208    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.372220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.372216    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d846076-2d4e-42fe-8909-b6fde688f282-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.372220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.372226    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzjwg\" (UniqueName: \"kubernetes.io/projected/5d846076-2d4e-42fe-8909-b6fde688f282-kube-api-access-lzjwg\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.372592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.372235    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d846076-2d4e-42fe-8909-b6fde688f282-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:59.461115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.461083    4925 generic.go:356] "Generic (PLEG): container finished" podID="088a7b18-ab88-4099-a815-99c6f57835ed" containerID="2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f" exitCode=0
Oct 30 19:48:59.461487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.461120    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"
Oct 30 19:48:59.461487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.461160    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" event={"ID":"088a7b18-ab88-4099-a815-99c6f57835ed","Type":"ContainerDied","Data":"2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f"}
Oct 30 19:48:59.461487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.461216    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd" event={"ID":"088a7b18-ab88-4099-a815-99c6f57835ed","Type":"ContainerDied","Data":"0188eafa4d17cf4516f953529ad09740c6d228e77f0f8ba26b270ce6341e03bf"}
Oct 30 19:48:59.461487 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.461249    4925 scope.go:122] "RemoveContainer" containerID="2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f"
Oct 30 19:48:59.463704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.463668    4925 generic.go:356] "Generic (PLEG): container finished" podID="5d846076-2d4e-42fe-8909-b6fde688f282" containerID="3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621" exitCode=0
Oct 30 19:48:59.463829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.463764    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"
Oct 30 19:48:59.463829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.463732    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" event={"ID":"5d846076-2d4e-42fe-8909-b6fde688f282","Type":"ContainerDied","Data":"3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621"}
Oct 30 19:48:59.463829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.463821    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5" event={"ID":"5d846076-2d4e-42fe-8909-b6fde688f282","Type":"ContainerDied","Data":"f787779dc573826652e29562e12bfdb900b84d3fb651a5982af91f65a59d8276"}
Oct 30 19:48:59.475391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.475350    4925 scope.go:122] "RemoveContainer" containerID="2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f"
Oct 30 19:48:59.475937 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:59.475915    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f\": container with ID starting with 2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f not found: ID does not exist" containerID="2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f"
Oct 30 19:48:59.478770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.475946    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f"} err="failed to get container status \"2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f\": rpc error: code = NotFound desc = could not find container \"2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f\": container with ID starting with 2280fa2c676bcecad5eb6db6b87659361330e7c2314131ecbbe0642b62b9155f not found: ID does not exist"
Oct 30 19:48:59.478914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.478898    4925 scope.go:122] "RemoveContainer" containerID="3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621"
Oct 30 19:48:59.491057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.491037    4925 scope.go:122] "RemoveContainer" containerID="b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3"
Oct 30 19:48:59.492017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.491986    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"]
Oct 30 19:48:59.498544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.498520    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-dclrd"]
Oct 30 19:48:59.502045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.502028    4925 scope.go:122] "RemoveContainer" containerID="3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621"
Oct 30 19:48:59.502342 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:59.502316    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621\": container with ID starting with 3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621 not found: ID does not exist" containerID="3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621"
Oct 30 19:48:59.502405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.502348    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621"} err="failed to get container status \"3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621\": rpc error: code = NotFound desc = could not find container \"3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621\": container with ID starting with 3c355e033ed00b31f09a6b48a9558e8a2a1242089485e50583b306c2021fc621 not found: ID does not exist"
Oct 30 19:48:59.502405 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.502364    4925 scope.go:122] "RemoveContainer" containerID="b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3"
Oct 30 19:48:59.502760 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:48:59.502680    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3\": container with ID starting with b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3 not found: ID does not exist" containerID="b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3"
Oct 30 19:48:59.502814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.502777    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3"} err="failed to get container status \"b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3\": rpc error: code = NotFound desc = could not find container \"b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3\": container with ID starting with b383b63d402eafd4ef384d4180c12ab4ae3cb02b472f274cc014a5cd64687db3 not found: ID does not exist"
Oct 30 19:48:59.516925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.516903    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"]
Oct 30 19:48:59.520526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.520505    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ddb6ea-openshift-default-c489869-qvxf5"]
Oct 30 19:48:59.591020 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.590987    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"]
Oct 30 19:48:59.591437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591423    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b4f0210a-e252-4aa6-8875-68a82b9a51d0" containerName="revoke-keys"
Oct 30 19:48:59.591493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591438    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f0210a-e252-4aa6-8875-68a82b9a51d0" containerName="revoke-keys"
Oct 30 19:48:59.591493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591463    4925 cpu_manager.go:414] "containerMap: removing container" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy"
Oct 30 19:48:59.591493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591468    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8411f4a9-165d-47a5-a255-e6d07376731e" containerName="istio-proxy"
Oct 30 19:48:59.591493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591476    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy"
Oct 30 19:48:59.591493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591480    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy"
Oct 30 19:48:59.591744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591502    4925 cpu_manager.go:414] "containerMap: removing container" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy"
Oct 30 19:48:59.591744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591506    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy"
Oct 30 19:48:59.591744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591580    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy"
Oct 30 19:48:59.591744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591587    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4afab18-0b83-4db6-a92e-3ad1c5a50da9" containerName="istio-proxy"
Oct 30 19:48:59.591744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591599    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" containerName="istio-proxy"
Oct 30 19:48:59.591744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591611    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy"
Oct 30 19:48:59.591744 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.591619    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f0210a-e252-4aa6-8875-68a82b9a51d0" containerName="revoke-keys"
Oct 30 19:48:59.596623 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.596603    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.604275 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.604253    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"]
Oct 30 19:48:59.605542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.605515    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:48:59.605656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.605581    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:48:59.605656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.605594    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:59.605656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.605602    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:48:59.681100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.681072    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6745af58-hfbqn\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.681269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.681132    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6745af58-hfbqn\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.681269 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.681216    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5sr\" (UniqueName: \"kubernetes.io/projected/9b53c969-8255-4df9-a6e8-830073a10b91-kube-api-access-js5sr\") pod \"maas-api-revoke-sub-6745af58-hfbqn\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.728777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.728746    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:59.781853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.781745    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6745af58-hfbqn\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.781853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.781808    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5sr\" (UniqueName: \"kubernetes.io/projected/9b53c969-8255-4df9-a6e8-830073a10b91-kube-api-access-js5sr\") pod \"maas-api-revoke-sub-6745af58-hfbqn\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.782160 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.781902    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6745af58-hfbqn\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.782550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.782529    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6745af58-hfbqn\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.784280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.784261    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6745af58-hfbqn\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.794024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.794005    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5sr\" (UniqueName: \"kubernetes.io/projected/9b53c969-8255-4df9-a6e8-830073a10b91-kube-api-access-js5sr\") pod \"maas-api-revoke-sub-6745af58-hfbqn\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:48:59.899846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.899805    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:59.912647 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:48:59.912623    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:49:00.193365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:00.193342    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"]
Oct 30 19:49:00.195839 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:49:00.195807    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b53c969_8255_4df9_a6e8_830073a10b91.slice/crio-ed13a5ffbce0599024d68b6c373124b04027bc9328dee0c2eeb5bb191344e00f WatchSource:0}: Error finding container ed13a5ffbce0599024d68b6c373124b04027bc9328dee0c2eeb5bb191344e00f: Status 404 returned error can't find the container with id ed13a5ffbce0599024d68b6c373124b04027bc9328dee0c2eeb5bb191344e00f
Oct 30 19:49:00.474298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:00.474266    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn" event={"ID":"9b53c969-8255-4df9-a6e8-830073a10b91","Type":"ContainerStarted","Data":"ed13a5ffbce0599024d68b6c373124b04027bc9328dee0c2eeb5bb191344e00f"}
Oct 30 19:49:00.548505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:00.548465    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:00.707348 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:00.707311    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:01.243660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:01.243630    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:01.262873 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:01.262845    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088a7b18-ab88-4099-a815-99c6f57835ed" path="/var/lib/kubelet/pods/088a7b18-ab88-4099-a815-99c6f57835ed/volumes"
Oct 30 19:49:01.263305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:01.263292    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" path="/var/lib/kubelet/pods/5d846076-2d4e-42fe-8909-b6fde688f282/volumes"
Oct 30 19:49:01.483579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:01.483546    4925 generic.go:356] "Generic (PLEG): container finished" podID="9b53c969-8255-4df9-a6e8-830073a10b91" containerID="7c0202327c8897b11c3531895b459198ec45da096aab80290e1ab70bd2ca5317" exitCode=0
Oct 30 19:49:01.483579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:01.483595    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn" event={"ID":"9b53c969-8255-4df9-a6e8-830073a10b91","Type":"ContainerDied","Data":"7c0202327c8897b11c3531895b459198ec45da096aab80290e1ab70bd2ca5317"}
Oct 30 19:49:01.854734 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:01.854598    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:02.252991 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.252951    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8" podUID="72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" containerName="payload-pre-processing" containerID="cri-o://c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1" gracePeriod=25
Oct 30 19:49:02.571193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.571159    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:02.640978 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.640954    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:49:02.710544 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.710516    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-openshift-ca\") pod \"9b53c969-8255-4df9-a6e8-830073a10b91\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") "
Oct 30 19:49:02.710754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.710592    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b53c969-8255-4df9-a6e8-830073a10b91-kube-api-access-js5sr\" (UniqueName: \"kubernetes.io/projected/9b53c969-8255-4df9-a6e8-830073a10b91-kube-api-access-js5sr\") pod \"9b53c969-8255-4df9-a6e8-830073a10b91\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") "
Oct 30 19:49:02.710754 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.710616    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-xks-ca\") pod \"9b53c969-8255-4df9-a6e8-830073a10b91\" (UID: \"9b53c969-8255-4df9-a6e8-830073a10b91\") "
Oct 30 19:49:02.710985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.710958    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-openshift-ca" pod "9b53c969-8255-4df9-a6e8-830073a10b91" (UID: "9b53c969-8255-4df9-a6e8-830073a10b91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:02.712913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.712893    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-xks-ca" pod "9b53c969-8255-4df9-a6e8-830073a10b91" (UID: "9b53c969-8255-4df9-a6e8-830073a10b91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:02.713072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.713057    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b53c969-8255-4df9-a6e8-830073a10b91-kube-api-access-js5sr" pod "9b53c969-8255-4df9-a6e8-830073a10b91" (UID: "9b53c969-8255-4df9-a6e8-830073a10b91"). InnerVolumeSpecName "kube-api-access-js5sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:02.812556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.812471    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:02.812556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.812501    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js5sr\" (UniqueName: \"kubernetes.io/projected/9b53c969-8255-4df9-a6e8-830073a10b91-kube-api-access-js5sr\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:02.812556 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:02.812513    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b53c969-8255-4df9-a6e8-830073a10b91-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:03.177019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:03.176983    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:03.499441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:03.499363    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"
Oct 30 19:49:03.499441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:03.499358    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn" event={"ID":"9b53c969-8255-4df9-a6e8-830073a10b91","Type":"ContainerDied","Data":"ed13a5ffbce0599024d68b6c373124b04027bc9328dee0c2eeb5bb191344e00f"}
Oct 30 19:49:03.499638 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:03.499458    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed13a5ffbce0599024d68b6c373124b04027bc9328dee0c2eeb5bb191344e00f"
Oct 30 19:49:03.884225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:03.884128    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:04.496772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:04.496730    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:04.714121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:04.714085    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:04.885050 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:04.884973    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:05.163655 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:05.163625    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:05.696360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:05.696308    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:05.944995 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:05.944960    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:06.035663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:06.035568    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"]
Oct 30 19:49:06.037589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:06.037563    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-fd7c77-hk8dl"]
Oct 30 19:49:06.228099 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:06.228068    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:06.388723 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:06.388616    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:06.631778 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:06.631746    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:06.844609 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:06.844576    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:07.079695 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.079660    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main" containerID="cri-o://9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698" gracePeriod=45
Oct 30 19:49:07.095739 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.095705    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" containerID="cri-o://d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72" gracePeriod=45
Oct 30 19:49:07.102072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.102047    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main" containerID="cri-o://01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af" gracePeriod=45
Oct 30 19:49:07.110158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.110132    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main" containerID="cri-o://c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d" gracePeriod=45
Oct 30 19:49:07.268060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.268017    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f0210a-e252-4aa6-8875-68a82b9a51d0" path="/var/lib/kubelet/pods/b4f0210a-e252-4aa6-8875-68a82b9a51d0/volumes"
Oct 30 19:49:07.406550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.406521    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:49:07.408568 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.408551    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:49:07.437522 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.437496    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:49:07.445085 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.444962    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:49:07.530736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.530668    4925 generic.go:356] "Generic (PLEG): container finished" podID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerID="d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72" exitCode=0
Oct 30 19:49:07.530736 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.530722    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"
Oct 30 19:49:07.530992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.530720    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" event={"ID":"e1ab7137-f18d-4dca-ae03-34a20e2fda7f","Type":"ContainerDied","Data":"d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72"}
Oct 30 19:49:07.530992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.530873    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" event={"ID":"e1ab7137-f18d-4dca-ae03-34a20e2fda7f","Type":"ContainerDied","Data":"34e549ba9bebd2cb556109f22d97bfc7a86fe0ce3ac3c0c18fa70e871807fc63"}
Oct 30 19:49:07.530992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.530902    4925 scope.go:122] "RemoveContainer" containerID="d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72"
Oct 30 19:49:07.533068 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.533042    4925 generic.go:356] "Generic (PLEG): container finished" podID="66623a2e-1062-4b84-99fe-7af3b038c072" containerID="c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d" exitCode=0
Oct 30 19:49:07.533194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.533088    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"
Oct 30 19:49:07.533194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.533144    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" event={"ID":"66623a2e-1062-4b84-99fe-7af3b038c072","Type":"ContainerDied","Data":"c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d"}
Oct 30 19:49:07.533324 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.533208    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9" event={"ID":"66623a2e-1062-4b84-99fe-7af3b038c072","Type":"ContainerDied","Data":"765c09c5aa986fb4c260da122126dae8a37f72b4ffe2d298946e11808284d91e"}
Oct 30 19:49:07.535088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.535067    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:07.535365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.535346    4925 generic.go:356] "Generic (PLEG): container finished" podID="c548c971-fe34-451d-a52d-73bbb9326e57" containerID="9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698" exitCode=0
Oct 30 19:49:07.535621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.535599    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"
Oct 30 19:49:07.535747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.535710    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" event={"ID":"c548c971-fe34-451d-a52d-73bbb9326e57","Type":"ContainerDied","Data":"9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698"}
Oct 30 19:49:07.535747 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.535741    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft" event={"ID":"c548c971-fe34-451d-a52d-73bbb9326e57","Type":"ContainerDied","Data":"b6d7430a648883bc2b3f1c325a84e86576734feeb032e4dac8f7a18c353452df"}
Oct 30 19:49:07.537478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.537436    4925 generic.go:356] "Generic (PLEG): container finished" podID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerID="01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af" exitCode=0
Oct 30 19:49:07.537575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.537507    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" event={"ID":"a3234780-a8fa-4d2c-8717-01d5d186506f","Type":"ContainerDied","Data":"01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af"}
Oct 30 19:49:07.537575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.537543    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85" event={"ID":"a3234780-a8fa-4d2c-8717-01d5d186506f","Type":"ContainerDied","Data":"59350e01da421c4697a4ccd147959b0f2cc492680c7fb54d2eb6b636d852c530"}
Oct 30 19:49:07.537658 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.537601    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"
Oct 30 19:49:07.543049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.543028    4925 scope.go:122] "RemoveContainer" containerID="d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72"
Oct 30 19:49:07.543317 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:49:07.543296    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72\": container with ID starting with d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72 not found: ID does not exist" containerID="d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72"
Oct 30 19:49:07.543437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.543321    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72"} err="failed to get container status \"d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72\": rpc error: code = NotFound desc = could not find container \"d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72\": container with ID starting with d924915fa7ae00efd6d2c67da1d9dc0ef0bcf0b61a05b05852e409cd9f82ce72 not found: ID does not exist"
Oct 30 19:49:07.543437 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.543338    4925 scope.go:122] "RemoveContainer" containerID="c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d"
Oct 30 19:49:07.552924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.552905    4925 scope.go:122] "RemoveContainer" containerID="c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d"
Oct 30 19:49:07.553236 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:49:07.553218    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d\": container with ID starting with c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d not found: ID does not exist" containerID="c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d"
Oct 30 19:49:07.553286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.553246    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d"} err="failed to get container status \"c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d\": rpc error: code = NotFound desc = could not find container \"c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d\": container with ID starting with c97e1faddac0638323acdc4a905d40b997e5562de9961b37264c49bc8776585d not found: ID does not exist"
Oct 30 19:49:07.553286 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.553265    4925 scope.go:122] "RemoveContainer" containerID="9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698"
Oct 30 19:49:07.562413 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.562393    4925 scope.go:122] "RemoveContainer" containerID="9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698"
Oct 30 19:49:07.562741 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:49:07.562720    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698\": container with ID starting with 9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698 not found: ID does not exist" containerID="9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698"
Oct 30 19:49:07.562815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.562747    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698"} err="failed to get container status \"9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698\": rpc error: code = NotFound desc = could not find container \"9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698\": container with ID starting with 9bd45914b5b779ca042d08d3f755e1b52d0221acf69996d6f2041c1a2caa4698 not found: ID does not exist"
Oct 30 19:49:07.562815 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.562762    4925 scope.go:122] "RemoveContainer" containerID="01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af"
Oct 30 19:49:07.569014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.568991    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-model-cache\") pod \"a3234780-a8fa-4d2c-8717-01d5d186506f\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") "
Oct 30 19:49:07.569105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569028    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-home\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-home\") pod \"a3234780-a8fa-4d2c-8717-01d5d186506f\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") "
Oct 30 19:49:07.569105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569057    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-model-cache\") pod \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") "
Oct 30 19:49:07.569105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569086    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-home\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-home\") pod \"c548c971-fe34-451d-a52d-73bbb9326e57\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") "
Oct 30 19:49:07.569246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569131    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-kube-api-access-2tqjl\" (UniqueName: \"kubernetes.io/projected/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-kube-api-access-2tqjl\") pod \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") "
Oct 30 19:49:07.569246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569179    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-home\") pod \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") "
Oct 30 19:49:07.569246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569214    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-tmp-dir\") pod \"66623a2e-1062-4b84-99fe-7af3b038c072\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") "
Oct 30 19:49:07.569419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569259    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66623a2e-1062-4b84-99fe-7af3b038c072-kube-api-access-mvg6g\" (UniqueName: \"kubernetes.io/projected/66623a2e-1062-4b84-99fe-7af3b038c072-kube-api-access-mvg6g\") pod \"66623a2e-1062-4b84-99fe-7af3b038c072\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") "
Oct 30 19:49:07.569419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569278    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-model-cache" pod "a3234780-a8fa-4d2c-8717-01d5d186506f" (UID: "a3234780-a8fa-4d2c-8717-01d5d186506f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.569419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569289    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-model-cache\") pod \"c548c971-fe34-451d-a52d-73bbb9326e57\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") "
Oct 30 19:49:07.569419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569291    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-home" pod "c548c971-fe34-451d-a52d-73bbb9326e57" (UID: "c548c971-fe34-451d-a52d-73bbb9326e57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.569419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569295    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-home" pod "a3234780-a8fa-4d2c-8717-01d5d186506f" (UID: "a3234780-a8fa-4d2c-8717-01d5d186506f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.569419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569320    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-dshm\") pod \"a3234780-a8fa-4d2c-8717-01d5d186506f\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") "
Oct 30 19:49:07.569419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569366    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3234780-a8fa-4d2c-8717-01d5d186506f-kube-api-access-tlzwl\" (UniqueName: \"kubernetes.io/projected/a3234780-a8fa-4d2c-8717-01d5d186506f-kube-api-access-tlzwl\") pod \"a3234780-a8fa-4d2c-8717-01d5d186506f\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") "
Oct 30 19:49:07.569419 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569395    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-dshm\") pod \"66623a2e-1062-4b84-99fe-7af3b038c072\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569452    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tls-certs\") pod \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569485    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66623a2e-1062-4b84-99fe-7af3b038c072-tls-certs\" (UniqueName: \"kubernetes.io/secret/66623a2e-1062-4b84-99fe-7af3b038c072-tls-certs\") pod \"66623a2e-1062-4b84-99fe-7af3b038c072\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569492    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-tmp-dir" pod "66623a2e-1062-4b84-99fe-7af3b038c072" (UID: "66623a2e-1062-4b84-99fe-7af3b038c072"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569529    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-home" pod "e1ab7137-f18d-4dca-ae03-34a20e2fda7f" (UID: "e1ab7137-f18d-4dca-ae03-34a20e2fda7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569534    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3234780-a8fa-4d2c-8717-01d5d186506f-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3234780-a8fa-4d2c-8717-01d5d186506f-tls-certs\") pod \"a3234780-a8fa-4d2c-8717-01d5d186506f\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569590    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tmp-dir\") pod \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569620    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-tmp-dir\") pod \"a3234780-a8fa-4d2c-8717-01d5d186506f\" (UID: \"a3234780-a8fa-4d2c-8717-01d5d186506f\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569654    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-dshm\") pod \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\" (UID: \"e1ab7137-f18d-4dca-ae03-34a20e2fda7f\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569711    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-dshm\") pod \"c548c971-fe34-451d-a52d-73bbb9326e57\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569759    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-model-cache\") pod \"66623a2e-1062-4b84-99fe-7af3b038c072\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569800    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-home\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-home\") pod \"66623a2e-1062-4b84-99fe-7af3b038c072\" (UID: \"66623a2e-1062-4b84-99fe-7af3b038c072\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569836    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c548c971-fe34-451d-a52d-73bbb9326e57-kube-api-access-p7vvk\" (UniqueName: \"kubernetes.io/projected/c548c971-fe34-451d-a52d-73bbb9326e57-kube-api-access-p7vvk\") pod \"c548c971-fe34-451d-a52d-73bbb9326e57\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569871    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c548c971-fe34-451d-a52d-73bbb9326e57-tls-certs\" (UniqueName: \"kubernetes.io/secret/c548c971-fe34-451d-a52d-73bbb9326e57-tls-certs\") pod \"c548c971-fe34-451d-a52d-73bbb9326e57\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") "
Oct 30 19:49:07.569980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.569931    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-tmp-dir\") pod \"c548c971-fe34-451d-a52d-73bbb9326e57\" (UID: \"c548c971-fe34-451d-a52d-73bbb9326e57\") "
Oct 30 19:49:07.571014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.570373    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.571014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.570397    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.571014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.570411    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.571014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.570440    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.571014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.570454    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.571014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.570644    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-model-cache" pod "c548c971-fe34-451d-a52d-73bbb9326e57" (UID: "c548c971-fe34-451d-a52d-73bbb9326e57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.571014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.570718    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-tmp-dir" pod "c548c971-fe34-451d-a52d-73bbb9326e57" (UID: "c548c971-fe34-451d-a52d-73bbb9326e57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.571014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.570824    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tmp-dir" pod "e1ab7137-f18d-4dca-ae03-34a20e2fda7f" (UID: "e1ab7137-f18d-4dca-ae03-34a20e2fda7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.571014 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.571015    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-home" pod "66623a2e-1062-4b84-99fe-7af3b038c072" (UID: "66623a2e-1062-4b84-99fe-7af3b038c072"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.571496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.571158    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-model-cache" pod "e1ab7137-f18d-4dca-ae03-34a20e2fda7f" (UID: "e1ab7137-f18d-4dca-ae03-34a20e2fda7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.571496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.571184    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-model-cache" pod "66623a2e-1062-4b84-99fe-7af3b038c072" (UID: "66623a2e-1062-4b84-99fe-7af3b038c072"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.571496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.571338    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-tmp-dir" pod "a3234780-a8fa-4d2c-8717-01d5d186506f" (UID: "a3234780-a8fa-4d2c-8717-01d5d186506f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.573062 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.573034    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66623a2e-1062-4b84-99fe-7af3b038c072-tls-certs" pod "66623a2e-1062-4b84-99fe-7af3b038c072" (UID: "66623a2e-1062-4b84-99fe-7af3b038c072"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:07.575423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.575393    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c548c971-fe34-451d-a52d-73bbb9326e57-kube-api-access-p7vvk" pod "c548c971-fe34-451d-a52d-73bbb9326e57" (UID: "c548c971-fe34-451d-a52d-73bbb9326e57"). InnerVolumeSpecName "kube-api-access-p7vvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:07.575817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.575761    4925 scope.go:122] "RemoveContainer" containerID="01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af"
Oct 30 19:49:07.575911 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.575896    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-dshm" pod "c548c971-fe34-451d-a52d-73bbb9326e57" (UID: "c548c971-fe34-451d-a52d-73bbb9326e57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.575983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.575918    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-kube-api-access-2tqjl" pod "e1ab7137-f18d-4dca-ae03-34a20e2fda7f" (UID: "e1ab7137-f18d-4dca-ae03-34a20e2fda7f"). InnerVolumeSpecName "kube-api-access-2tqjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:07.575983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.575919    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-dshm" pod "a3234780-a8fa-4d2c-8717-01d5d186506f" (UID: "a3234780-a8fa-4d2c-8717-01d5d186506f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.575983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.575919    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-dshm" pod "66623a2e-1062-4b84-99fe-7af3b038c072" (UID: "66623a2e-1062-4b84-99fe-7af3b038c072"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.575983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.575958    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66623a2e-1062-4b84-99fe-7af3b038c072-kube-api-access-mvg6g" pod "66623a2e-1062-4b84-99fe-7af3b038c072" (UID: "66623a2e-1062-4b84-99fe-7af3b038c072"). InnerVolumeSpecName "kube-api-access-mvg6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:07.575983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.575980    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tls-certs" pod "e1ab7137-f18d-4dca-ae03-34a20e2fda7f" (UID: "e1ab7137-f18d-4dca-ae03-34a20e2fda7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:07.576172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.575982    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3234780-a8fa-4d2c-8717-01d5d186506f-kube-api-access-tlzwl" pod "a3234780-a8fa-4d2c-8717-01d5d186506f" (UID: "a3234780-a8fa-4d2c-8717-01d5d186506f"). InnerVolumeSpecName "kube-api-access-tlzwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:07.576172 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:49:07.576068    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af\": container with ID starting with 01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af not found: ID does not exist" containerID="01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af"
Oct 30 19:49:07.576172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.576091    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-dshm" pod "e1ab7137-f18d-4dca-ae03-34a20e2fda7f" (UID: "e1ab7137-f18d-4dca-ae03-34a20e2fda7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:07.576172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.576093    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af"} err="failed to get container status \"01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af\": rpc error: code = NotFound desc = could not find container \"01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af\": container with ID starting with 01b09cccf6136cdfaa765fcab33c8492ca8389469ec4a02f5627216bad62b8af not found: ID does not exist"
Oct 30 19:49:07.576432 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.576417    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3234780-a8fa-4d2c-8717-01d5d186506f-tls-certs" pod "a3234780-a8fa-4d2c-8717-01d5d186506f" (UID: "a3234780-a8fa-4d2c-8717-01d5d186506f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:07.577192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.577177    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c548c971-fe34-451d-a52d-73bbb9326e57-tls-certs" pod "c548c971-fe34-451d-a52d-73bbb9326e57" (UID: "c548c971-fe34-451d-a52d-73bbb9326e57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:07.671093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671062    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671093 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671092    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tqjl\" (UniqueName: \"kubernetes.io/projected/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-kube-api-access-2tqjl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671103    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvg6g\" (UniqueName: \"kubernetes.io/projected/66623a2e-1062-4b84-99fe-7af3b038c072-kube-api-access-mvg6g\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671112    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671122    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671131    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzwl\" (UniqueName: \"kubernetes.io/projected/a3234780-a8fa-4d2c-8717-01d5d186506f-kube-api-access-tlzwl\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671140    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671148    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671156    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66623a2e-1062-4b84-99fe-7af3b038c072-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671165    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3234780-a8fa-4d2c-8717-01d5d186506f-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671173    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671181    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3234780-a8fa-4d2c-8717-01d5d186506f-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671188    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab7137-f18d-4dca-ae03-34a20e2fda7f-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671196    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671203    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671210    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66623a2e-1062-4b84-99fe-7af3b038c072-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671218    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7vvk\" (UniqueName: \"kubernetes.io/projected/c548c971-fe34-451d-a52d-73bbb9326e57-kube-api-access-p7vvk\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671225    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c548c971-fe34-451d-a52d-73bbb9326e57-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.671373 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.671233    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c548c971-fe34-451d-a52d-73bbb9326e57-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:07.691955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.691927    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8" podUID="72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:49:07.854927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.854902    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"]
Oct 30 19:49:07.862433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.862410    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg"]
Oct 30 19:49:07.873766 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.873745    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"]
Oct 30 19:49:07.876134 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.876112    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/simulator-w2-39b867-kserve-86cbcdb7c4-cn7j9"]
Oct 30 19:49:07.889170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.889140    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"]
Oct 30 19:49:07.895573 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.895550    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-2-w2-39b867-kserve-576dc667d5-fg6ft"]
Oct 30 19:49:07.907659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.907617    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"]
Oct 30 19:49:07.909394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:07.909373    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39b867/distinct-w2-39b867-kserve-7fc56f95bb-xfb85"]
Oct 30 19:49:08.244158 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:08.244126    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:08.343800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:08.343721    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39b867/premium-w2-39b867-kserve-7ff487f55f-s9lgg" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:08.888108 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:08.888071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:09.261468 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:09.261432    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" path="/var/lib/kubelet/pods/66623a2e-1062-4b84-99fe-7af3b038c072/volumes"
Oct 30 19:49:09.261876 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:09.261861    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" path="/var/lib/kubelet/pods/a3234780-a8fa-4d2c-8717-01d5d186506f/volumes"
Oct 30 19:49:09.262235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:09.262223    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" path="/var/lib/kubelet/pods/c548c971-fe34-451d-a52d-73bbb9326e57/volumes"
Oct 30 19:49:09.262904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:09.262890    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" path="/var/lib/kubelet/pods/e1ab7137-f18d-4dca-ae03-34a20e2fda7f/volumes"
Oct 30 19:49:09.661594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:09.661562    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:10.272404 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:10.272368    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:11.050325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:11.050280    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:11.463196 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:11.463167    4925 scope.go:122] "RemoveContainer" containerID="c3590a594be029f8bccd357b96ab2c91ca0ab0ff6622c70862e4d799655c42da"
Oct 30 19:49:11.472987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:11.472964    4925 scope.go:122] "RemoveContainer" containerID="79c9d95c1bc9c92bd675ca3c42231e7a32e34aa314703b813ab35f0f33b81d6d"
Oct 30 19:49:11.481772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:11.481755    4925 scope.go:122] "RemoveContainer" containerID="75feea487f5696f0ff344459c293d6800ce920baae130083b12cc0018cd21133"
Oct 30 19:49:11.759771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:11.759668    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:12.261870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:12.261840    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:12.864575 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:12.864548    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:13.384877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:13.384835    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:13.540667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:13.540633    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:13.699667 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:13.699619    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:13.959217 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:13.959133    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:14.147075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:14.147037    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:14.330578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:14.330487    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:14.441172 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:14.441129    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"]
Oct 30 19:49:14.442069 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:14.442040    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" containerID="cri-o://264ef4b3f41c2f8c52d00efd5175a6cc90a515e3dd365171d40e9d5444725275" gracePeriod=30
Oct 30 19:49:14.454721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:14.454675    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"]
Oct 30 19:49:14.457898 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:14.457874    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy" containerID="cri-o://daf390c4986197c06ea34f1cb050a64218a7e1892d5e0cef107e658e275c51bc" gracePeriod=30
Oct 30 19:49:14.552440 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:14.551812    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:14.781120 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:14.781078    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:15.002847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.002813    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:15.081905 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.081831    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"]
Oct 30 19:49:15.082251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082230    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main"
Oct 30 19:49:15.082294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082253    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main"
Oct 30 19:49:15.082294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082274    4925 cpu_manager.go:414] "containerMap: removing container" podUID="9b53c969-8255-4df9-a6e8-830073a10b91" containerName="revoke-keys"
Oct 30 19:49:15.082294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082279    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b53c969-8255-4df9-a6e8-830073a10b91" containerName="revoke-keys"
Oct 30 19:49:15.082294 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082292    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082297    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082305    4925 cpu_manager.go:414] "containerMap: removing container" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082309    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082321    4925 cpu_manager.go:414] "containerMap: removing container" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082326    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082331    4925 cpu_manager.go:414] "containerMap: removing container" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082336    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d846076-2d4e-42fe-8909-b6fde688f282" containerName="istio-proxy"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082392    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b53c969-8255-4df9-a6e8-830073a10b91" containerName="revoke-keys"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082402    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ab7137-f18d-4dca-ae03-34a20e2fda7f" containerName="main"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082409    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3234780-a8fa-4d2c-8717-01d5d186506f" containerName="main"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082415    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="66623a2e-1062-4b84-99fe-7af3b038c072" containerName="main"
Oct 30 19:49:15.082439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.082426    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c548c971-fe34-451d-a52d-73bbb9326e57" containerName="main"
Oct 30 19:49:15.088350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.088335    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.091412 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.091384    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:15.091578 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.091507    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:49:15.091663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.091642    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:49:15.091886 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.091660    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:49:15.093962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.093938    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"]
Oct 30 19:49:15.169088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.169057    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.169265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.169109    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.169375 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.169255    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqnk\" (UniqueName: \"kubernetes.io/projected/48fb4328-7802-4fb9-a9af-82b4914edbca-kube-api-access-qsqnk\") pod \"maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.270473 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.270436    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqnk\" (UniqueName: \"kubernetes.io/projected/48fb4328-7802-4fb9-a9af-82b4914edbca-kube-api-access-qsqnk\") pod \"maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.270643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.270508    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.270643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.270568    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.271239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.271218    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.272997 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.272980    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.281817 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.281791    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqnk\" (UniqueName: \"kubernetes.io/projected/48fb4328-7802-4fb9-a9af-82b4914edbca-kube-api-access-qsqnk\") pod \"maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.404956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.404928    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:15.667514 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.667442    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:15.686624 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:15.686596    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"]
Oct 30 19:49:15.689772 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:49:15.689743    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fb4328_7802_4fb9_a9af_82b4914edbca.slice/crio-2687702b294104c48328e37305a55ada7d481770d58e669dcb62003c63d9af52 WatchSource:0}: Error finding container 2687702b294104c48328e37305a55ada7d481770d58e669dcb62003c63d9af52: Status 404 returned error can't find the container with id 2687702b294104c48328e37305a55ada7d481770d58e669dcb62003c63d9af52
Oct 30 19:49:16.492429 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:16.492392    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:16.602478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:16.602448    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4" event={"ID":"48fb4328-7802-4fb9-a9af-82b4914edbca","Type":"ContainerStarted","Data":"51c579f2ab27757514ec74ce5da5327cbb8dae2c149c3cb147b4fed4e0513257"}
Oct 30 19:49:16.602648 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:16.602498    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4" event={"ID":"48fb4328-7802-4fb9-a9af-82b4914edbca","Type":"ContainerStarted","Data":"2687702b294104c48328e37305a55ada7d481770d58e669dcb62003c63d9af52"}
Oct 30 19:49:16.620963 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:16.620921    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4" podStartSLOduration=1.6209078099999998 podStartE2EDuration="1.62090781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:16.619175655 +0000 UTC m=+2591.522188692" watchObservedRunningTime="2026-10-30 19:49:16.62090781 +0000 UTC m=+2591.523920846"
Oct 30 19:49:17.086828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:17.086791    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:17.609669 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:17.609635    4925 generic.go:356] "Generic (PLEG): container finished" podID="48fb4328-7802-4fb9-a9af-82b4914edbca" containerID="51c579f2ab27757514ec74ce5da5327cbb8dae2c149c3cb147b4fed4e0513257" exitCode=0
Oct 30 19:49:17.610141 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:17.609718    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4" event={"ID":"48fb4328-7802-4fb9-a9af-82b4914edbca","Type":"ContainerDied","Data":"51c579f2ab27757514ec74ce5da5327cbb8dae2c149c3cb147b4fed4e0513257"}
Oct 30 19:49:17.692266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:17.692231    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8" podUID="72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:49:18.035114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.035084    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"]
Oct 30 19:49:18.040064 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.040039    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda5464d-s56gm"]
Oct 30 19:49:18.345306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.345219    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:18.392733 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.392677    4925 patch_prober.go:29] interesting pod/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:18.392902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.392758    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:18.747243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.747217    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:18.805881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.805852    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-xks-ca\") pod \"48fb4328-7802-4fb9-a9af-82b4914edbca\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") "
Oct 30 19:49:18.806058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.805895    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-openshift-ca\") pod \"48fb4328-7802-4fb9-a9af-82b4914edbca\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") "
Oct 30 19:49:18.806058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.805992    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fb4328-7802-4fb9-a9af-82b4914edbca-kube-api-access-qsqnk\" (UniqueName: \"kubernetes.io/projected/48fb4328-7802-4fb9-a9af-82b4914edbca-kube-api-access-qsqnk\") pod \"48fb4328-7802-4fb9-a9af-82b4914edbca\" (UID: \"48fb4328-7802-4fb9-a9af-82b4914edbca\") "
Oct 30 19:49:18.806298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.806276    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-openshift-ca" pod "48fb4328-7802-4fb9-a9af-82b4914edbca" (UID: "48fb4328-7802-4fb9-a9af-82b4914edbca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:18.808179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.808157    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-xks-ca" pod "48fb4328-7802-4fb9-a9af-82b4914edbca" (UID: "48fb4328-7802-4fb9-a9af-82b4914edbca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:18.808272 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.808218    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fb4328-7802-4fb9-a9af-82b4914edbca-kube-api-access-qsqnk" pod "48fb4328-7802-4fb9-a9af-82b4914edbca" (UID: "48fb4328-7802-4fb9-a9af-82b4914edbca"). InnerVolumeSpecName "kube-api-access-qsqnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:18.859848 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.859817    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:18.907156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.907128    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:18.907156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.907153    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb4328-7802-4fb9-a9af-82b4914edbca-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:18.907325 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:18.907165    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsqnk\" (UniqueName: \"kubernetes.io/projected/48fb4328-7802-4fb9-a9af-82b4914edbca-kube-api-access-qsqnk\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.261049 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.260974    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f" path="/var/lib/kubelet/pods/e7518ab0-ddaf-40ae-9c6e-8647f5b2f82f/volumes"
Oct 30 19:49:19.465750 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.465675    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:19.625324 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.625294    4925 generic.go:356] "Generic (PLEG): container finished" podID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerID="264ef4b3f41c2f8c52d00efd5175a6cc90a515e3dd365171d40e9d5444725275" exitCode=0
Oct 30 19:49:19.625484 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.625443    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" event={"ID":"550e4d4e-8cea-4f60-8061-2c5276c6e800","Type":"ContainerDied","Data":"264ef4b3f41c2f8c52d00efd5175a6cc90a515e3dd365171d40e9d5444725275"}
Oct 30 19:49:19.627289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.627258    4925 generic.go:356] "Generic (PLEG): container finished" podID="c3301076-3aef-468a-a79a-04c0ca49231c" containerID="daf390c4986197c06ea34f1cb050a64218a7e1892d5e0cef107e658e275c51bc" exitCode=0
Oct 30 19:49:19.627438 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.627421    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" event={"ID":"c3301076-3aef-468a-a79a-04c0ca49231c","Type":"ContainerDied","Data":"daf390c4986197c06ea34f1cb050a64218a7e1892d5e0cef107e658e275c51bc"}
Oct 30 19:49:19.628796 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.628775    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4" event={"ID":"48fb4328-7802-4fb9-a9af-82b4914edbca","Type":"ContainerDied","Data":"2687702b294104c48328e37305a55ada7d481770d58e669dcb62003c63d9af52"}
Oct 30 19:49:19.628883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.628811    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2687702b294104c48328e37305a55ada7d481770d58e669dcb62003c63d9af52"
Oct 30 19:49:19.628883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.628865    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"
Oct 30 19:49:19.705603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.705577    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:49:19.766345 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.766318    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:49:19.815577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815513    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-credential-socket\") pod \"c3301076-3aef-468a-a79a-04c0ca49231c\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") "
Oct 30 19:49:19.815577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815551    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-kube-api-access-8rb4d\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-kube-api-access-8rb4d\") pod \"c3301076-3aef-468a-a79a-04c0ca49231c\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") "
Oct 30 19:49:19.815577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815576    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-data\") pod \"550e4d4e-8cea-4f60-8061-2c5276c6e800\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") "
Oct 30 19:49:19.815837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815595    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-kube-api-access-dcvl8\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-kube-api-access-dcvl8\") pod \"550e4d4e-8cea-4f60-8061-2c5276c6e800\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") "
Oct 30 19:49:19.815837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815623    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-token\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-token\") pod \"550e4d4e-8cea-4f60-8061-2c5276c6e800\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") "
Oct 30 19:49:19.815837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815643    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-certs\") pod \"550e4d4e-8cea-4f60-8061-2c5276c6e800\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") "
Oct 30 19:49:19.815837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815677    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-envoy\") pod \"550e4d4e-8cea-4f60-8061-2c5276c6e800\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") "
Oct 30 19:49:19.815837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815777    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-podinfo\") pod \"550e4d4e-8cea-4f60-8061-2c5276c6e800\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") "
Oct 30 19:49:19.816109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815868    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-credential-socket" pod "c3301076-3aef-468a-a79a-04c0ca49231c" (UID: "c3301076-3aef-468a-a79a-04c0ca49231c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.816109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.815868    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-data" pod "550e4d4e-8cea-4f60-8061-2c5276c6e800" (UID: "550e4d4e-8cea-4f60-8061-2c5276c6e800"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.816109 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816020    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-certs" pod "550e4d4e-8cea-4f60-8061-2c5276c6e800" (UID: "550e4d4e-8cea-4f60-8061-2c5276c6e800"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.816239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816149    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-socket\") pod \"550e4d4e-8cea-4f60-8061-2c5276c6e800\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") "
Oct 30 19:49:19.816239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816196    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-data\") pod \"c3301076-3aef-468a-a79a-04c0ca49231c\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") "
Oct 30 19:49:19.816354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816247    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-istio-token\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-istio-token\") pod \"c3301076-3aef-468a-a79a-04c0ca49231c\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") "
Oct 30 19:49:19.816354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816283    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-credential-socket\") pod \"550e4d4e-8cea-4f60-8061-2c5276c6e800\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") "
Oct 30 19:49:19.816354 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816323    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-socket\") pod \"c3301076-3aef-468a-a79a-04c0ca49231c\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") "
Oct 30 19:49:19.816704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816532    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-envoy\") pod \"c3301076-3aef-468a-a79a-04c0ca49231c\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") "
Oct 30 19:49:19.816704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816559    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550e4d4e-8cea-4f60-8061-2c5276c6e800-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e4d4e-8cea-4f60-8061-2c5276c6e800-istiod-ca-cert\") pod \"550e4d4e-8cea-4f60-8061-2c5276c6e800\" (UID: \"550e4d4e-8cea-4f60-8061-2c5276c6e800\") "
Oct 30 19:49:19.816704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816589    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-certs\") pod \"c3301076-3aef-468a-a79a-04c0ca49231c\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") "
Oct 30 19:49:19.816704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816625    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3301076-3aef-468a-a79a-04c0ca49231c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3301076-3aef-468a-a79a-04c0ca49231c-istio-podinfo\") pod \"c3301076-3aef-468a-a79a-04c0ca49231c\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") "
Oct 30 19:49:19.816704 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816661    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3301076-3aef-468a-a79a-04c0ca49231c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3301076-3aef-468a-a79a-04c0ca49231c-istiod-ca-cert\") pod \"c3301076-3aef-468a-a79a-04c0ca49231c\" (UID: \"c3301076-3aef-468a-a79a-04c0ca49231c\") "
Oct 30 19:49:19.817075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.817053    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.817152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.817083    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.817152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.817098    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.818834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.816326    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-socket" pod "550e4d4e-8cea-4f60-8061-2c5276c6e800" (UID: "550e4d4e-8cea-4f60-8061-2c5276c6e800"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.818834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.817412    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3301076-3aef-468a-a79a-04c0ca49231c-istiod-ca-cert" pod "c3301076-3aef-468a-a79a-04c0ca49231c" (UID: "c3301076-3aef-468a-a79a-04c0ca49231c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:19.818834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.817764    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550e4d4e-8cea-4f60-8061-2c5276c6e800-istiod-ca-cert" pod "550e4d4e-8cea-4f60-8061-2c5276c6e800" (UID: "550e4d4e-8cea-4f60-8061-2c5276c6e800"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:19.818834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.818037    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-certs" pod "c3301076-3aef-468a-a79a-04c0ca49231c" (UID: "c3301076-3aef-468a-a79a-04c0ca49231c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.818834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.818189    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-podinfo" pod "550e4d4e-8cea-4f60-8061-2c5276c6e800" (UID: "550e4d4e-8cea-4f60-8061-2c5276c6e800"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:19.818834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.818380    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-data" pod "c3301076-3aef-468a-a79a-04c0ca49231c" (UID: "c3301076-3aef-468a-a79a-04c0ca49231c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.818834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.818536    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-socket" pod "c3301076-3aef-468a-a79a-04c0ca49231c" (UID: "c3301076-3aef-468a-a79a-04c0ca49231c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.818834 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.818720    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-credential-socket" pod "550e4d4e-8cea-4f60-8061-2c5276c6e800" (UID: "550e4d4e-8cea-4f60-8061-2c5276c6e800"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.819303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.818881    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-kube-api-access-dcvl8" pod "550e4d4e-8cea-4f60-8061-2c5276c6e800" (UID: "550e4d4e-8cea-4f60-8061-2c5276c6e800"). InnerVolumeSpecName "kube-api-access-dcvl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.819303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.819169    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-kube-api-access-8rb4d" pod "c3301076-3aef-468a-a79a-04c0ca49231c" (UID: "c3301076-3aef-468a-a79a-04c0ca49231c"). InnerVolumeSpecName "kube-api-access-8rb4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.819753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.819579    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-envoy" pod "550e4d4e-8cea-4f60-8061-2c5276c6e800" (UID: "550e4d4e-8cea-4f60-8061-2c5276c6e800"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.820031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.819804    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-envoy" pod "c3301076-3aef-468a-a79a-04c0ca49231c" (UID: "c3301076-3aef-468a-a79a-04c0ca49231c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:19.820892 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.820856    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-token" pod "550e4d4e-8cea-4f60-8061-2c5276c6e800" (UID: "550e4d4e-8cea-4f60-8061-2c5276c6e800"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.820999 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.820948    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3301076-3aef-468a-a79a-04c0ca49231c-istio-podinfo" pod "c3301076-3aef-468a-a79a-04c0ca49231c" (UID: "c3301076-3aef-468a-a79a-04c0ca49231c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:19.822176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.822145    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-istio-token" pod "c3301076-3aef-468a-a79a-04c0ca49231c" (UID: "c3301076-3aef-468a-a79a-04c0ca49231c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:19.918567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918535    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918559    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918567 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918570    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918579    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918589    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/550e4d4e-8cea-4f60-8061-2c5276c6e800-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918597    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918605    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3301076-3aef-468a-a79a-04c0ca49231c-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918613    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3301076-3aef-468a-a79a-04c0ca49231c-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918621    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rb4d\" (UniqueName: \"kubernetes.io/projected/c3301076-3aef-468a-a79a-04c0ca49231c-kube-api-access-8rb4d\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918630    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcvl8\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-kube-api-access-dcvl8\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918637    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918646    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918656    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/550e4d4e-8cea-4f60-8061-2c5276c6e800-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918664    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/550e4d4e-8cea-4f60-8061-2c5276c6e800-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:19.918910 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:19.918672    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3301076-3aef-468a-a79a-04c0ca49231c-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:20.064901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.064866    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:20.229228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.229197    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:20.635956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.635875    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"
Oct 30 19:49:20.635956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.635910    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp" event={"ID":"550e4d4e-8cea-4f60-8061-2c5276c6e800","Type":"ContainerDied","Data":"4c1595422483d06911cb8b4919e56c7e10ad921a295010ec93ebdef350d41855"}
Oct 30 19:49:20.636202 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.635967    4925 scope.go:122] "RemoveContainer" containerID="264ef4b3f41c2f8c52d00efd5175a6cc90a515e3dd365171d40e9d5444725275"
Oct 30 19:49:20.637594 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.637575    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"
Oct 30 19:49:20.637721 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.637580    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg" event={"ID":"c3301076-3aef-468a-a79a-04c0ca49231c","Type":"ContainerDied","Data":"5c5e6451ccb877ca60150cb53b74a58f83deeb53b649bd2d87c4326fa93324ff"}
Oct 30 19:49:20.647229 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.647140    4925 scope.go:122] "RemoveContainer" containerID="daf390c4986197c06ea34f1cb050a64218a7e1892d5e0cef107e658e275c51bc"
Oct 30 19:49:20.660591 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.660568    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"]
Oct 30 19:49:20.664850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.664828    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-bwslp"]
Oct 30 19:49:20.674972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.674951    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"]
Oct 30 19:49:20.679181 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.679163    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fd7c77-openshift-default-6995885647-vjnxg"]
Oct 30 19:49:20.946431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:20.946401    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:21.260461 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:21.260379    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" path="/var/lib/kubelet/pods/550e4d4e-8cea-4f60-8061-2c5276c6e800/volumes"
Oct 30 19:49:21.260872 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:21.260856    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" path="/var/lib/kubelet/pods/c3301076-3aef-468a-a79a-04c0ca49231c/volumes"
Oct 30 19:49:21.561942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:21.561847    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:21.975123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:21.975071    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:22.126319 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:49:22.126286    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:49:22.134392 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.134363    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:22.288765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.288731    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"]
Oct 30 19:49:22.289242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.289211    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy" containerID="cri-o://92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881" gracePeriod=30
Oct 30 19:49:22.330987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.330952    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:22.532469 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.532443    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:49:22.547506 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.547479    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:22.645245 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.645209    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-plugins-config-volume\") pod \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\" (UID: \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\") "
Oct 30 19:49:22.645446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.645254    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-kube-api-access-dxmsh\" (UniqueName: \"kubernetes.io/projected/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-kube-api-access-dxmsh\") pod \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\" (UID: \"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1\") "
Oct 30 19:49:22.645579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.645557    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-plugins-config-volume" pod "72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" (UID: "72ee03ca-995d-4c4a-85c2-f7b28a1e40f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:22.647634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.647604    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-kube-api-access-dxmsh" pod "72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" (UID: "72ee03ca-995d-4c4a-85c2-f7b28a1e40f1"). InnerVolumeSpecName "kube-api-access-dxmsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:22.653467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.653442    4925 generic.go:356] "Generic (PLEG): container finished" podID="72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" containerID="c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1" exitCode=0
Oct 30 19:49:22.653587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.653497    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8" event={"ID":"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1","Type":"ContainerDied","Data":"c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1"}
Oct 30 19:49:22.653587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.653538    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8" event={"ID":"72ee03ca-995d-4c4a-85c2-f7b28a1e40f1","Type":"ContainerDied","Data":"8ef60d69a16bbff1a2400217ccaef5b78cd56c10e49d779aa7d6592a419d83e8"}
Oct 30 19:49:22.653587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.653546    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"
Oct 30 19:49:22.653587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.653559    4925 scope.go:122] "RemoveContainer" containerID="c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1"
Oct 30 19:49:22.664075 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.664059    4925 scope.go:122] "RemoveContainer" containerID="c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1"
Oct 30 19:49:22.664326 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:49:22.664310    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1\": container with ID starting with c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1 not found: ID does not exist" containerID="c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1"
Oct 30 19:49:22.664383 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.664334    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1"} err="failed to get container status \"c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1\": rpc error: code = NotFound desc = could not find container \"c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1\": container with ID starting with c474720bb6a4ea679b59e59a84a1b3dcf05fa398f19faceaaecac499606651e1 not found: ID does not exist"
Oct 30 19:49:22.678615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.678592    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"]
Oct 30 19:49:22.684595 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.684574    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-39b867-5b4c4b4bf9-kxww8"]
Oct 30 19:49:22.746422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.746393    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:22.746422 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:22.746420    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxmsh\" (UniqueName: \"kubernetes.io/projected/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1-kube-api-access-dxmsh\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:23.062150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:23.062116    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:23.261376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:23.261344    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" path="/var/lib/kubelet/pods/72ee03ca-995d-4c4a-85c2-f7b28a1e40f1/volumes"
Oct 30 19:49:23.760185 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:23.760147    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:24.553414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:24.553381    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:25.151266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:25.151217    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:25.336902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:25.336860    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:25.503816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:25.503731    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:25.815799 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:25.815715    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:26.201914 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.201885    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"]
Oct 30 19:49:26.202293 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202279    4925 cpu_manager.go:414] "containerMap: removing container" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy"
Oct 30 19:49:26.202346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202294    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy"
Oct 30 19:49:26.202346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202313    4925 cpu_manager.go:414] "containerMap: removing container" podUID="72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" containerName="payload-pre-processing"
Oct 30 19:49:26.202346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202322    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" containerName="payload-pre-processing"
Oct 30 19:49:26.202346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202340    4925 cpu_manager.go:414] "containerMap: removing container" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy"
Oct 30 19:49:26.202346 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202345    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy"
Oct 30 19:49:26.202507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202353    4925 cpu_manager.go:414] "containerMap: removing container" podUID="48fb4328-7802-4fb9-a9af-82b4914edbca" containerName="revoke-keys"
Oct 30 19:49:26.202507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202358    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fb4328-7802-4fb9-a9af-82b4914edbca" containerName="revoke-keys"
Oct 30 19:49:26.202507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202423    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="550e4d4e-8cea-4f60-8061-2c5276c6e800" containerName="istio-proxy"
Oct 30 19:49:26.202507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202434    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fb4328-7802-4fb9-a9af-82b4914edbca" containerName="revoke-keys"
Oct 30 19:49:26.202507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202443    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3301076-3aef-468a-a79a-04c0ca49231c" containerName="istio-proxy"
Oct 30 19:49:26.202507 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.202451    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ee03ca-995d-4c4a-85c2-f7b28a1e40f1" containerName="payload-pre-processing"
Oct 30 19:49:26.206855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.206838    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.214190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.213806    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:49:26.214190 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.213934    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 19:49:26.214418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.214207    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:26.216823 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.216798    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:49:26.217330 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.217309    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"]
Oct 30 19:49:26.278323 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.278296    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83e1c4c-sxrk9\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.278456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.278374    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83e1c4c-sxrk9\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.278456 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.278418    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2867\" (UniqueName: \"kubernetes.io/projected/1a8dba39-f3f4-4560-b0aa-72577b8312a4-kube-api-access-q2867\") pod \"maas-api-revoke-sub-a83e1c4c-sxrk9\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.379858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.379826    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83e1c4c-sxrk9\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.379858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.379858    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2867\" (UniqueName: \"kubernetes.io/projected/1a8dba39-f3f4-4560-b0aa-72577b8312a4-kube-api-access-q2867\") pod \"maas-api-revoke-sub-a83e1c4c-sxrk9\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.380079 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.379923    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83e1c4c-sxrk9\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.380495 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.380472    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83e1c4c-sxrk9\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.382322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.382303    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83e1c4c-sxrk9\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.391646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.391625    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2867\" (UniqueName: \"kubernetes.io/projected/1a8dba39-f3f4-4560-b0aa-72577b8312a4-kube-api-access-q2867\") pod \"maas-api-revoke-sub-a83e1c4c-sxrk9\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.522239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.522177    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:26.802332 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:26.802307    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"]
Oct 30 19:49:26.805251 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:49:26.805224    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8dba39_f3f4_4560_b0aa_72577b8312a4.slice/crio-4539ad7e7052cc84f6c8a36fe4e2b9636367c9da94da58dba5a907490ed252b6 WatchSource:0}: Error finding container 4539ad7e7052cc84f6c8a36fe4e2b9636367c9da94da58dba5a907490ed252b6: Status 404 returned error can't find the container with id 4539ad7e7052cc84f6c8a36fe4e2b9636367c9da94da58dba5a907490ed252b6
Oct 30 19:49:27.541844 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.541818    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:49:27.590672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.590642    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-socket\") pod \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") "
Oct 30 19:49:27.590858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.590758    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istiod-ca-cert\") pod \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") "
Oct 30 19:49:27.590858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.590795    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-data\") pod \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") "
Oct 30 19:49:27.590858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.590814    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-envoy\") pod \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") "
Oct 30 19:49:27.591058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.590953    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-podinfo\") pod \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") "
Oct 30 19:49:27.591058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591005    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-credential-socket\") pod \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") "
Oct 30 19:49:27.591058 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591052    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-kube-api-access-5m268\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-kube-api-access-5m268\") pod \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") "
Oct 30 19:49:27.591242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591085    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-certs\") pod \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") "
Oct 30 19:49:27.591242 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591111    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-token\") pod \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\" (UID: \"f2c5d294-364e-42fe-85dc-dce17f1a31ea\") "
Oct 30 19:49:27.592087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.590950    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-socket" pod "f2c5d294-364e-42fe-85dc-dce17f1a31ea" (UID: "f2c5d294-364e-42fe-85dc-dce17f1a31ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.592087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591110    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-data" pod "f2c5d294-364e-42fe-85dc-dce17f1a31ea" (UID: "f2c5d294-364e-42fe-85dc-dce17f1a31ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.592087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591170    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istiod-ca-cert" pod "f2c5d294-364e-42fe-85dc-dce17f1a31ea" (UID: "f2c5d294-364e-42fe-85dc-dce17f1a31ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:27.592087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591476    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-credential-socket" pod "f2c5d294-364e-42fe-85dc-dce17f1a31ea" (UID: "f2c5d294-364e-42fe-85dc-dce17f1a31ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.592087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591745    4925 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-credential-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.592087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591768    4925 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-socket\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.592087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591785    4925 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istiod-ca-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.592087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591798    4925 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-data\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.592087 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.591889    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-certs" pod "f2c5d294-364e-42fe-85dc-dce17f1a31ea" (UID: "f2c5d294-364e-42fe-85dc-dce17f1a31ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.593277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.593255    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-podinfo" pod "f2c5d294-364e-42fe-85dc-dce17f1a31ea" (UID: "f2c5d294-364e-42fe-85dc-dce17f1a31ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:27.593548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.593529    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-envoy" pod "f2c5d294-364e-42fe-85dc-dce17f1a31ea" (UID: "f2c5d294-364e-42fe-85dc-dce17f1a31ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:27.593987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.593667    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-kube-api-access-5m268" pod "f2c5d294-364e-42fe-85dc-dce17f1a31ea" (UID: "f2c5d294-364e-42fe-85dc-dce17f1a31ea"). InnerVolumeSpecName "kube-api-access-5m268". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:27.594200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.594184    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-token" pod "f2c5d294-364e-42fe-85dc-dce17f1a31ea" (UID: "f2c5d294-364e-42fe-85dc-dce17f1a31ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:27.692737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.692707    4925 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-envoy\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.692737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.692734    4925 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-podinfo\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.692737 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.692743    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m268\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-kube-api-access-5m268\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.692989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.692753    4925 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d294-364e-42fe-85dc-dce17f1a31ea-workload-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.692989 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.692762    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5d294-364e-42fe-85dc-dce17f1a31ea-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:27.696100 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.696075    4925 generic.go:356] "Generic (PLEG): container finished" podID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerID="92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881" exitCode=0
Oct 30 19:49:27.696247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.696146    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"
Oct 30 19:49:27.696247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.696163    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" event={"ID":"f2c5d294-364e-42fe-85dc-dce17f1a31ea","Type":"ContainerDied","Data":"92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881"}
Oct 30 19:49:27.696247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.696226    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l" event={"ID":"f2c5d294-364e-42fe-85dc-dce17f1a31ea","Type":"ContainerDied","Data":"02b56bc9e00f17a006a74e7a32ac44d087823512b2956d215e8ae9dc1c46f530"}
Oct 30 19:49:27.696428 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.696258    4925 scope.go:122] "RemoveContainer" containerID="92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881"
Oct 30 19:49:27.698936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.698487    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9" event={"ID":"1a8dba39-f3f4-4560-b0aa-72577b8312a4","Type":"ContainerStarted","Data":"6ad9bbc107c83781c1bf076fd56a1ad1b74cf232536485c1cdb70dec49dba919"}
Oct 30 19:49:27.698936 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.698534    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9" event={"ID":"1a8dba39-f3f4-4560-b0aa-72577b8312a4","Type":"ContainerStarted","Data":"4539ad7e7052cc84f6c8a36fe4e2b9636367c9da94da58dba5a907490ed252b6"}
Oct 30 19:49:27.707767 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.707749    4925 scope.go:122] "RemoveContainer" containerID="c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e"
Oct 30 19:49:27.716534 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.716480    4925 scope.go:122] "RemoveContainer" containerID="92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881"
Oct 30 19:49:27.717037 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:49:27.717008    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881\": container with ID starting with 92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881 not found: ID does not exist" containerID="92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881"
Oct 30 19:49:27.717037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.717048    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881"} err="failed to get container status \"92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881\": rpc error: code = NotFound desc = could not find container \"92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881\": container with ID starting with 92ca4d9c8480a011ed6faf162a9dd4bbd68ce6e730ef274f3bbf7dc5e25cc881 not found: ID does not exist"
Oct 30 19:49:27.717240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.717072    4925 scope.go:122] "RemoveContainer" containerID="c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e"
Oct 30 19:49:27.717407 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:49:27.717393    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e\": container with ID starting with c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e not found: ID does not exist" containerID="c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e"
Oct 30 19:49:27.717462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.717413    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e"} err="failed to get container status \"c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e\": rpc error: code = NotFound desc = could not find container \"c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e\": container with ID starting with c8dca14f713b6fd34ffcb3a667bde3536b67a7f92d1c4853c32b53501b9e640e not found: ID does not exist"
Oct 30 19:49:27.717822 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.717783    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9" podStartSLOduration=1.717771511 podStartE2EDuration="1.717771511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:27.71583703 +0000 UTC m=+2602.618850066" watchObservedRunningTime="2026-10-30 19:49:27.717771511 +0000 UTC m=+2602.620784547"
Oct 30 19:49:27.730128 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.730104    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"]
Oct 30 19:49:27.734756 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:27.734735    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39b867-openshift-default-756f9d477f-4sh2l"]
Oct 30 19:49:28.710755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:28.710717    4925 generic.go:356] "Generic (PLEG): container finished" podID="1a8dba39-f3f4-4560-b0aa-72577b8312a4" containerID="6ad9bbc107c83781c1bf076fd56a1ad1b74cf232536485c1cdb70dec49dba919" exitCode=0
Oct 30 19:49:28.711213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:28.710798    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9" event={"ID":"1a8dba39-f3f4-4560-b0aa-72577b8312a4","Type":"ContainerDied","Data":"6ad9bbc107c83781c1bf076fd56a1ad1b74cf232536485c1cdb70dec49dba919"}
Oct 30 19:49:28.842526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:28.842489    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:29.013777 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.013668    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:29.188971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.188940    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:29.260802 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.260764    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" path="/var/lib/kubelet/pods/f2c5d294-364e-42fe-85dc-dce17f1a31ea/volumes"
Oct 30 19:49:29.352048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.351950    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:29.865941 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.865915    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:29.916394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.916360    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8dba39-f3f4-4560-b0aa-72577b8312a4-kube-api-access-q2867\" (UniqueName: \"kubernetes.io/projected/1a8dba39-f3f4-4560-b0aa-72577b8312a4-kube-api-access-q2867\") pod \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") "
Oct 30 19:49:29.916563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.916483    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-openshift-ca\") pod \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") "
Oct 30 19:49:29.916563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.916530    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-xks-ca\") pod \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\" (UID: \"1a8dba39-f3f4-4560-b0aa-72577b8312a4\") "
Oct 30 19:49:29.916943 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.916904    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-openshift-ca" pod "1a8dba39-f3f4-4560-b0aa-72577b8312a4" (UID: "1a8dba39-f3f4-4560-b0aa-72577b8312a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:29.918642 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.918618    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8dba39-f3f4-4560-b0aa-72577b8312a4-kube-api-access-q2867" pod "1a8dba39-f3f4-4560-b0aa-72577b8312a4" (UID: "1a8dba39-f3f4-4560-b0aa-72577b8312a4"). InnerVolumeSpecName "kube-api-access-q2867". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:29.918803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:29.918776    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-xks-ca" pod "1a8dba39-f3f4-4560-b0aa-72577b8312a4" (UID: "1a8dba39-f3f4-4560-b0aa-72577b8312a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:30.018083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:30.018009    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.018083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:30.018037    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8dba39-f3f4-4560-b0aa-72577b8312a4-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.018083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:30.018047    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2867\" (UniqueName: \"kubernetes.io/projected/1a8dba39-f3f4-4560-b0aa-72577b8312a4-kube-api-access-q2867\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:30.724904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:30.724874    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"
Oct 30 19:49:30.725110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:30.724871    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9" event={"ID":"1a8dba39-f3f4-4560-b0aa-72577b8312a4","Type":"ContainerDied","Data":"4539ad7e7052cc84f6c8a36fe4e2b9636367c9da94da58dba5a907490ed252b6"}
Oct 30 19:49:30.725110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:30.725016    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4539ad7e7052cc84f6c8a36fe4e2b9636367c9da94da58dba5a907490ed252b6"
Oct 30 19:49:31.713008 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:31.712972    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:37.960615 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:37.960578    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:41.364385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:41.364329    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:41.504466 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:41.504428    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"]
Oct 30 19:49:41.508868 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:41.508835    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-17e904-nklk4"]
Oct 30 19:49:41.571651 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:41.571618    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:41.743948 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:41.743909    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:42.100516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:42.100425    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:43.260542 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:43.260508    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fb4328-7802-4fb9-a9af-82b4914edbca" path="/var/lib/kubelet/pods/48fb4328-7802-4fb9-a9af-82b4914edbca/volumes"
Oct 30 19:49:48.753927 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:48.753897    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:48.949056 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:48.949015    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:49.169870 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:49.169834    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:52.127868 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:49:52.127827    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:49:57.200306 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:57.200265    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:57.370971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:57.370941    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:58.162076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:58.162038    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:58.679753 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:49:58.679718    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:04.232987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:04.232950    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:11.603398 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.603363    4925 scope.go:122] "RemoveContainer" containerID="79b99d254880be2c8557890b2235eb304ce610c513641554b738db6c8bdcc815"
Oct 30 19:50:11.613136 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.613112    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"]
Oct 30 19:50:11.613481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.613469    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy"
Oct 30 19:50:11.613527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.613482    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy"
Oct 30 19:50:11.613527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.613505    4925 cpu_manager.go:414] "containerMap: removing container" podUID="1a8dba39-f3f4-4560-b0aa-72577b8312a4" containerName="revoke-keys"
Oct 30 19:50:11.613527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.613510    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8dba39-f3f4-4560-b0aa-72577b8312a4" containerName="revoke-keys"
Oct 30 19:50:11.613646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.613568    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy"
Oct 30 19:50:11.613646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.613577    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8dba39-f3f4-4560-b0aa-72577b8312a4" containerName="revoke-keys"
Oct 30 19:50:11.617072 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.617053    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.620526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.620502    4925 scope.go:122] "RemoveContainer" containerID="dd3ceeb3c68c6ef1c0c44c7ce099d941628d09f53d95c381e680f3b533cc7ad0"
Oct 30 19:50:11.631307 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.631282    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"]
Oct 30 19:50:11.632110 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.632090    4925 scope.go:122] "RemoveContainer" containerID="9d5e0441ba75da981c8f360537c1ae6c4df968e1dac071114bd0556f0467d027"
Oct 30 19:50:11.643288 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.643267    4925 scope.go:122] "RemoveContainer" containerID="e5172923bc93581339aaf4c0744dd24c1527fab78cee829f4c0fec390c8988b5"
Oct 30 19:50:11.695749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.695680    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.695858 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.695806    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.695945 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.695882    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.696006 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.695947    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.696060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.696045    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.696098 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.696089    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.696152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.696107    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.696220 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.696201    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhvrs\" (UniqueName: \"kubernetes.io/projected/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-kube-api-access-hhvrs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.696261 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.696231    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.796985    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797045    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797085    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797109    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797153 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797143    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhvrs\" (UniqueName: \"kubernetes.io/projected/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-kube-api-access-hhvrs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797171    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797216    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797384 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797241    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797565    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797657    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797724    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797771    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.797992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.797975    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.798071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.798051    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.799496 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.799471    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.799592 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.799580    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.809177 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.809155    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.809268 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.809216    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhvrs\" (UniqueName: \"kubernetes.io/projected/d3021cf9-99e2-4612-82c5-3c9bc2a39e30-kube-api-access-hhvrs\") pod \"maas-default-gateway-openshift-default-855f9574cf-vs4nx\" (UID: \"d3021cf9-99e2-4612-82c5-3c9bc2a39e30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:11.938877 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:11.938852    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:12.229418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:12.229385    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"]
Oct 30 19:50:12.231783 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:50:12.231672    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3021cf9_99e2_4612_82c5_3c9bc2a39e30.slice/crio-210acfe63742b8f42f8de74086ee79fce3d9341e1bbeed47d5f58f7a9c1c8fe6 WatchSource:0}: Error finding container 210acfe63742b8f42f8de74086ee79fce3d9341e1bbeed47d5f58f7a9c1c8fe6: Status 404 returned error can't find the container with id 210acfe63742b8f42f8de74086ee79fce3d9341e1bbeed47d5f58f7a9c1c8fe6
Oct 30 19:50:12.234579 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:12.234546    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:12.234710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:12.234610    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:12.234710 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:12.234642    4925 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:12.986352 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:12.986319    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" event={"ID":"d3021cf9-99e2-4612-82c5-3c9bc2a39e30","Type":"ContainerStarted","Data":"fa13de2349014c917d7f67d2ed4cf27a328ff1eb66a97e585074e83a930d4eec"}
Oct 30 19:50:12.986805 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:12.986372    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" event={"ID":"d3021cf9-99e2-4612-82c5-3c9bc2a39e30","Type":"ContainerStarted","Data":"210acfe63742b8f42f8de74086ee79fce3d9341e1bbeed47d5f58f7a9c1c8fe6"}
Oct 30 19:50:13.939903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:13.939870    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:13.941397 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:13.941375    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vs4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Oct 30 19:50:13.941458 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:13.941423    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" podUID="d3021cf9-99e2-4612-82c5-3c9bc2a39e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Oct 30 19:50:14.939761 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:14.939721    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vs4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Oct 30 19:50:14.940163 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:14.939782    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" podUID="d3021cf9-99e2-4612-82c5-3c9bc2a39e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Oct 30 19:50:15.940076 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:15.940033    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vs4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Oct 30 19:50:15.940498 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:15.940105    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" podUID="d3021cf9-99e2-4612-82c5-3c9bc2a39e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Oct 30 19:50:16.939213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:16.939174    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vs4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Oct 30 19:50:16.939387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:16.939250    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" podUID="d3021cf9-99e2-4612-82c5-3c9bc2a39e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Oct 30 19:50:17.940266 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:17.940231    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vs4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Oct 30 19:50:17.940797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:17.940287    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" podUID="d3021cf9-99e2-4612-82c5-3c9bc2a39e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Oct 30 19:50:18.939875 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:18.939832    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vs4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body=
Oct 30 19:50:18.940057 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:18.939913    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" podUID="d3021cf9-99e2-4612-82c5-3c9bc2a39e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused"
Oct 30 19:50:20.940532 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:20.940486    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vs4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:20.940963 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:20.940557    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" podUID="d3021cf9-99e2-4612-82c5-3c9bc2a39e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:21.981859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:21.981818    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-vs4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:21.982295 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:21.981924    4925 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" podUID="d3021cf9-99e2-4612-82c5-3c9bc2a39e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:22.132654 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:50:22.132610    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:50:22.508105 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:22.508078    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:22.508555 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:22.508536    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:22.515010 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:22.514993    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx"
Oct 30 19:50:22.532026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:22.531983    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-vs4nx" podStartSLOduration=11.531970642 podStartE2EDuration="11.531970642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:13.019077471 +0000 UTC m=+2647.922090509" watchObservedRunningTime="2026-10-30 19:50:22.531970642 +0000 UTC m=+2657.434983740"
Oct 30 19:50:25.847760 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:25.847721    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:27.557362 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:27.557331    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:52.125394 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:50:52.125353    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:50:58.733662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:50:58.733627    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:51:06.427483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:06.427457    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:51:06.454350 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:06.454328    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:51:11.697962 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:11.697931    4925 scope.go:122] "RemoveContainer" containerID="9621c6d3ac88554d4f9401d99a8819935bb8667c4bbeb5cf7af7cfb8fc7f3b2c"
Oct 30 19:51:11.709896 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:11.709866    4925 scope.go:122] "RemoveContainer" containerID="239cb62f5a3c106885e3c1933140c862caa73c882ef841e8e94d21374f079897"
Oct 30 19:51:11.720629 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:11.720610    4925 scope.go:122] "RemoveContainer" containerID="de14abf6fd37ffeb5c2c81bc3b9a8dba069874f586dcfbf6925501002cc13b7e"
Oct 30 19:51:17.921931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:17.921895    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:51:22.125731 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:51:22.125664    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:51:32.284853 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:32.284807    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-hcvdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:32.285381 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:32.284856    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podUID="68707443-c353-4c78-9d2f-fb9e915c6b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:39.032289 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:39.032244    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"]
Oct 30 19:51:39.035200 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:39.035163    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"]
Oct 30 19:51:39.037974 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:39.037946    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14afed5-r5ljk"]
Oct 30 19:51:39.039605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:39.039576    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a049a861-9x58t"]
Oct 30 19:51:39.260816 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:39.260786    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a" path="/var/lib/kubelet/pods/b42979bb-1e1e-4b6e-913d-5e6bbdcc7d4a/volumes"
Oct 30 19:51:39.261188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:39.261175    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d628f4ea-91c2-4e94-8c59-76dbf7eccfdf" path="/var/lib/kubelet/pods/d628f4ea-91c2-4e94-8c59-76dbf7eccfdf/volumes"
Oct 30 19:51:52.124871 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:51:52.124790    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:51:58.797606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:51:58.797560    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:52:01.026731 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:01.026682    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"]
Oct 30 19:52:01.031091 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:01.031068    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"]
Oct 30 19:52:01.033890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:01.033855    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-278885ed-8x2t7"]
Oct 30 19:52:01.035988 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:01.035961    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75c6cae-7zhnf"]
Oct 30 19:52:01.261596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:01.261564    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84cb409f-a07b-42d7-b58a-9913245b09a3" path="/var/lib/kubelet/pods/84cb409f-a07b-42d7-b58a-9913245b09a3/volumes"
Oct 30 19:52:01.262017 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:01.262003    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff5fa4b3-aa17-418a-a826-2fe9c22dfc50" path="/var/lib/kubelet/pods/ff5fa4b3-aa17-418a-a826-2fe9c22dfc50/volumes"
Oct 30 19:52:11.758121 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:11.758090    4925 scope.go:122] "RemoveContainer" containerID="d713baf9f082552aa43871b680c6142db05bbfa792db7cc117a7a46044bfc14b"
Oct 30 19:52:11.767601 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:11.767578    4925 scope.go:122] "RemoveContainer" containerID="77c69c6414edd471801ab235612bdecf710054b622f57df90d2d143c977b96a0"
Oct 30 19:52:11.775790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:11.775772    4925 scope.go:122] "RemoveContainer" containerID="1c72f18374dfdcdce8f816e328f1c18eb2f260529cb88543efbdb1b29335c58d"
Oct 30 19:52:11.783513 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:11.783497    4925 scope.go:122] "RemoveContainer" containerID="173d8955c4c829113a4e349d9031339f5c200cae26dede90dab2530a5671847c"
Oct 30 19:52:11.791237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:11.791220    4925 scope.go:122] "RemoveContainer" containerID="88025e997075cd9cd4a3bd5f466b234fee45c91fdb723341871196c038d3f9a4"
Oct 30 19:52:11.798909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:11.798892    4925 scope.go:122] "RemoveContainer" containerID="2324d400d21687ecc0b7aafe4ce1009ce1211bb03d0c1492b546e78ce1a78778"
Oct 30 19:52:11.806837 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:11.806814    4925 scope.go:122] "RemoveContainer" containerID="8003a2756d5b3103e336a4460e71cd2b09f0ae29a4824c5d1a08df6aaf3dfd54"
Oct 30 19:52:22.125512 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:52:22.125470    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:52:52.124760 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:52:52.124724    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:52:58.928712 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:52:58.927719    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:11.853209 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:11.853178    4925 scope.go:122] "RemoveContainer" containerID="270aaa28b637541ad2eb3472031a3b6666122143520c6f0430defcaa3f50e94f"
Oct 30 19:53:11.862234 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:11.862196    4925 scope.go:122] "RemoveContainer" containerID="5992ee929773159d6129f5a79408fa5869f80b398576e18bfc822997fc9ee90a"
Oct 30 19:53:11.870243 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:11.870224    4925 scope.go:122] "RemoveContainer" containerID="f30f7e2af36929b5a6c6bb0b9b6b5bc85495e953976ab1f70368622d7c3fde3b"
Oct 30 19:53:11.880656 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:11.880634    4925 scope.go:122] "RemoveContainer" containerID="10b3b9de7c58b08f6607027a9a745b8ada582c65125eb2bf58ec4bd2848ab6df"
Oct 30 19:53:11.888560 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:11.888538    4925 scope.go:122] "RemoveContainer" containerID="88e9a1f220e4cd25facb593b07a6948738c444b681dbdd854b3ef3067de4ef18"
Oct 30 19:53:11.896124 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:11.896104    4925 scope.go:122] "RemoveContainer" containerID="8e4d2d13c18f6cf451513a7befd2930196ffcb42ed4922bcc8fb967e2d344383"
Oct 30 19:53:11.903559 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:11.903542    4925 scope.go:122] "RemoveContainer" containerID="f5db5fc996447d831817d4918463509265431e2017da4a46af068b1b6b7cc366"
Oct 30 19:53:11.911116 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:11.911092    4925 scope.go:122] "RemoveContainer" containerID="a8c63f0e754eb67917282b21c02df09a201e8dcb90db3f36e5e7fe8b727d386a"
Oct 30 19:53:11.918326 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:11.918305    4925 scope.go:122] "RemoveContainer" containerID="109c74776b8b37c795007be56453c9e166dea81b9c53ce1afd7a9da31b6e2062"
Oct 30 19:53:17.286925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:17.286825    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-hcvdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:53:17.286925 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:17.286896    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podUID="68707443-c353-4c78-9d2f-fb9e915c6b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:22.124680 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:53:22.124641    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:53:22.206789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:22.206760    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:22.372213 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:22.372181    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:23.026360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:23.026320    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"]
Oct 30 19:53:23.033516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:23.033489    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ac1413b-kzjxz"]
Oct 30 19:53:23.035103 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:23.035079    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"]
Oct 30 19:53:23.036902 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:23.036879    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71362ddf-7pkcb"]
Oct 30 19:53:23.260786 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:23.260754    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a9c464-b6b7-4a31-8338-96afc7cb9388" path="/var/lib/kubelet/pods/96a9c464-b6b7-4a31-8338-96afc7cb9388/volumes"
Oct 30 19:53:23.261188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:23.261155    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b9c38b-cbd5-437d-be46-605b76970539" path="/var/lib/kubelet/pods/e6b9c38b-cbd5-437d-be46-605b76970539/volumes"
Oct 30 19:53:48.274762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:48.274729    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:48.438028 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:48.437996    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:49.852280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:49.852237    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:50.016643 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:50.016609    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:52.124738 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:53:52.124682    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:53:58.974516 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:53:58.974478    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:03.028509 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:03.028476    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"]
Oct 30 19:54:03.033680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:03.033654    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6745af58-hfbqn"]
Oct 30 19:54:03.260971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:03.260938    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b53c969-8255-4df9-a6e8-830073a10b91" path="/var/lib/kubelet/pods/9b53c969-8255-4df9-a6e8-830073a10b91/volumes"
Oct 30 19:54:07.982855 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:07.982823    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:11.967313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:11.967281    4925 scope.go:122] "RemoveContainer" containerID="5092ec193d761c7c987cc6c3404060d765150405388f4381de1517f75f22ca64"
Oct 30 19:54:11.976489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:11.976467    4925 scope.go:122] "RemoveContainer" containerID="67c89b5660adaf87cdb02e0470fa7b95e260e720e700526fa9a4376f6e5d3026"
Oct 30 19:54:11.984238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:11.984206    4925 scope.go:122] "RemoveContainer" containerID="6d9c55f1ded7db86966ebb5cb6df652f44e077d5c02e99027e62d936f0ca0849"
Oct 30 19:54:11.991602 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:11.991583    4925 scope.go:122] "RemoveContainer" containerID="47f33450bc67d37408d5c8d9a3b4b4258912f4e284b92067524a59aca5a04bdf"
Oct 30 19:54:11.999400 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:11.999373    4925 scope.go:122] "RemoveContainer" containerID="a5c0a9fe7208fd26f6067c62815635174b121689cb49352a2f17e4eb41655a6c"
Oct 30 19:54:12.007047 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:12.007026    4925 scope.go:122] "RemoveContainer" containerID="e218325be94f2a91395c2a46120fe966a8a5b4785caaafe2ad110db82acc8df8"
Oct 30 19:54:12.014612 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:12.014595    4925 scope.go:122] "RemoveContainer" containerID="3f62c74b25700ac07f61735b5d87fdcef5ee1efde7a177096c05fc955bcdf1d5"
Oct 30 19:54:17.815367 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:17.815324    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"]
Oct 30 19:54:17.815922 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:17.815542    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl" podUID="94931795-ec51-464e-aeb7-104def6ebcd3" containerName="discovery" containerID="cri-o://4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e" gracePeriod=30
Oct 30 19:54:18.067563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.067490    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:54:18.153494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.153462    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-kubeconfig\") pod \"94931795-ec51-464e-aeb7-104def6ebcd3\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") "
Oct 30 19:54:18.153709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.153504    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94931795-ec51-464e-aeb7-104def6ebcd3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/94931795-ec51-464e-aeb7-104def6ebcd3-local-certs\") pod \"94931795-ec51-464e-aeb7-104def6ebcd3\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") "
Oct 30 19:54:18.153709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.153530    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-kube-api-access-26qh6\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-kube-api-access-26qh6\") pod \"94931795-ec51-464e-aeb7-104def6ebcd3\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") "
Oct 30 19:54:18.153709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.153570    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-istio-token\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-istio-token\") pod \"94931795-ec51-464e-aeb7-104def6ebcd3\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") "
Oct 30 19:54:18.153709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.153638    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-dns-cert\") pod \"94931795-ec51-464e-aeb7-104def6ebcd3\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") "
Oct 30 19:54:18.153982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.153714    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-ca-configmap\") pod \"94931795-ec51-464e-aeb7-104def6ebcd3\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") "
Oct 30 19:54:18.153982 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.153747    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-cacerts\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-cacerts\") pod \"94931795-ec51-464e-aeb7-104def6ebcd3\" (UID: \"94931795-ec51-464e-aeb7-104def6ebcd3\") "
Oct 30 19:54:18.154530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.154468    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-ca-configmap" pod "94931795-ec51-464e-aeb7-104def6ebcd3" (UID: "94931795-ec51-464e-aeb7-104def6ebcd3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:18.156071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.156035    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-istio-token" pod "94931795-ec51-464e-aeb7-104def6ebcd3" (UID: "94931795-ec51-464e-aeb7-104def6ebcd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:18.156298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.156267    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-cacerts" pod "94931795-ec51-464e-aeb7-104def6ebcd3" (UID: "94931795-ec51-464e-aeb7-104def6ebcd3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:18.156414 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.156352    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-kubeconfig" pod "94931795-ec51-464e-aeb7-104def6ebcd3" (UID: "94931795-ec51-464e-aeb7-104def6ebcd3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:18.156587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.156560    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94931795-ec51-464e-aeb7-104def6ebcd3-local-certs" pod "94931795-ec51-464e-aeb7-104def6ebcd3" (UID: "94931795-ec51-464e-aeb7-104def6ebcd3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:18.156587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.156565    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-kube-api-access-26qh6" pod "94931795-ec51-464e-aeb7-104def6ebcd3" (UID: "94931795-ec51-464e-aeb7-104def6ebcd3"). InnerVolumeSpecName "kube-api-access-26qh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:18.156587 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.156582    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-dns-cert" pod "94931795-ec51-464e-aeb7-104def6ebcd3" (UID: "94931795-ec51-464e-aeb7-104def6ebcd3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:18.255423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.255392    4925 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-istio-token\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:18.255423 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.255425    4925 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-dns-cert\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:18.255645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.255439    4925 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/94931795-ec51-464e-aeb7-104def6ebcd3-istio-csr-ca-configmap\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:18.255645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.255454    4925 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-cacerts\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:18.255645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.255465    4925 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94931795-ec51-464e-aeb7-104def6ebcd3-istio-kubeconfig\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:18.255645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.255476    4925 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/94931795-ec51-464e-aeb7-104def6ebcd3-local-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:18.255645 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.255489    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26qh6\" (UniqueName: \"kubernetes.io/projected/94931795-ec51-464e-aeb7-104def6ebcd3-kube-api-access-26qh6\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:18.640657 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.640618    4925 generic.go:356] "Generic (PLEG): container finished" podID="94931795-ec51-464e-aeb7-104def6ebcd3" containerID="4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e" exitCode=0
Oct 30 19:54:18.640887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.640669    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"
Oct 30 19:54:18.640887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.640722    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl" event={"ID":"94931795-ec51-464e-aeb7-104def6ebcd3","Type":"ContainerDied","Data":"4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e"}
Oct 30 19:54:18.640887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.640775    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl" event={"ID":"94931795-ec51-464e-aeb7-104def6ebcd3","Type":"ContainerDied","Data":"71fae5eaad5168c392b1b85857a13522583a19a7ce2dc692bbda498b39c89d74"}
Oct 30 19:54:18.640887 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.640794    4925 scope.go:122] "RemoveContainer" containerID="4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e"
Oct 30 19:54:18.653446 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.653424    4925 scope.go:122] "RemoveContainer" containerID="4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e"
Oct 30 19:54:18.653862 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:54:18.653833    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e\": container with ID starting with 4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e not found: ID does not exist" containerID="4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e"
Oct 30 19:54:18.653955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.653876    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e"} err="failed to get container status \"4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e\": rpc error: code = NotFound desc = could not find container \"4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e\": container with ID starting with 4625f6cf081b301afb30180344f37d29ce1b0821ee7a2a5eebbadb073094036e not found: ID does not exist"
Oct 30 19:54:18.664001 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.663980    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"]
Oct 30 19:54:18.667531 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:18.667510    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5j4cl"]
Oct 30 19:54:19.260765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:19.260733    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94931795-ec51-464e-aeb7-104def6ebcd3" path="/var/lib/kubelet/pods/94931795-ec51-464e-aeb7-104def6ebcd3/volumes"
Oct 30 19:54:22.123920 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:54:22.123874    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:54:27.931340 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:27.931305    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:28.520193 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:28.520159    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:28.700569 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:28.700527    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:30.025225 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:30.025191    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"]
Oct 30 19:54:30.030045 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:30.030018    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83e1c4c-sxrk9"]
Oct 30 19:54:31.260971 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:31.260939    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8dba39-f3f4-4560-b0aa-72577b8312a4" path="/var/lib/kubelet/pods/1a8dba39-f3f4-4560-b0aa-72577b8312a4/volumes"
Oct 30 19:54:31.721828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:31.721789    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:31.913762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:31.913710    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:32.884772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.884742    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:32.933913 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.933881    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"]
Oct 30 19:54:32.934246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.934225    4925 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy"
Oct 30 19:54:32.934365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.934247    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy"
Oct 30 19:54:32.934365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.934285    4925 cpu_manager.go:414] "containerMap: removing container" podUID="94931795-ec51-464e-aeb7-104def6ebcd3" containerName="discovery"
Oct 30 19:54:32.934365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.934290    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94931795-ec51-464e-aeb7-104def6ebcd3" containerName="discovery"
Oct 30 19:54:32.934365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.934341    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5d294-364e-42fe-85dc-dce17f1a31ea" containerName="istio-proxy"
Oct 30 19:54:32.934365 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.934354    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="94931795-ec51-464e-aeb7-104def6ebcd3" containerName="discovery"
Oct 30 19:54:32.938976 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.938958    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:32.946333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.946294    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a89c71\""
Oct 30 19:54:32.948416 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:32.948394    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"]
Oct 30 19:54:33.069738 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.069673    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:33.085361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.085323    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gm6d\" (UniqueName: \"kubernetes.io/projected/bf998829-ac74-49b1-b3a0-be4b456030fd-kube-api-access-6gm6d\") pod \"payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz\" (UID: \"bf998829-ac74-49b1-b3a0-be4b456030fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:33.085660 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.085632    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf998829-ac74-49b1-b3a0-be4b456030fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz\" (UID: \"bf998829-ac74-49b1-b3a0-be4b456030fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:33.186907 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.186870    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf998829-ac74-49b1-b3a0-be4b456030fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz\" (UID: \"bf998829-ac74-49b1-b3a0-be4b456030fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:33.187138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.187074    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gm6d\" (UniqueName: \"kubernetes.io/projected/bf998829-ac74-49b1-b3a0-be4b456030fd-kube-api-access-6gm6d\") pod \"payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz\" (UID: \"bf998829-ac74-49b1-b3a0-be4b456030fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:33.187830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.187804    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf998829-ac74-49b1-b3a0-be4b456030fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz\" (UID: \"bf998829-ac74-49b1-b3a0-be4b456030fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:33.203520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.203491    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gm6d\" (UniqueName: \"kubernetes.io/projected/bf998829-ac74-49b1-b3a0-be4b456030fd-kube-api-access-6gm6d\") pod \"payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz\" (UID: \"bf998829-ac74-49b1-b3a0-be4b456030fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:33.254520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.254486    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:33.531317 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.531109    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"]
Oct 30 19:54:33.534513 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:54:33.534473    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf998829_ac74_49b1_b3a0_be4b456030fd.slice/crio-2b851f334ef8cdbb9ea0dc3232091a16316e0dda6b4883065fda3f31b0a9a4f7 WatchSource:0}: Error finding container 2b851f334ef8cdbb9ea0dc3232091a16316e0dda6b4883065fda3f31b0a9a4f7: Status 404 returned error can't find the container with id 2b851f334ef8cdbb9ea0dc3232091a16316e0dda6b4883065fda3f31b0a9a4f7
Oct 30 19:54:33.718173 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.718141    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz" event={"ID":"bf998829-ac74-49b1-b3a0-be4b456030fd","Type":"ContainerStarted","Data":"6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce"}
Oct 30 19:54:33.718385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.718192    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz" event={"ID":"bf998829-ac74-49b1-b3a0-be4b456030fd","Type":"ContainerStarted","Data":"2b851f334ef8cdbb9ea0dc3232091a16316e0dda6b4883065fda3f31b0a9a4f7"}
Oct 30 19:54:33.718385 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.718282    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:33.780659 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.780585    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz" podStartSLOduration=1.7805639 podStartE2EDuration="1.7805639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:33.779920348 +0000 UTC m=+2908.682933397" watchObservedRunningTime="2026-10-30 19:54:33.7805639 +0000 UTC m=+2908.683576939"
Oct 30 19:54:33.870909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:33.870817    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:34.550869 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:34.550837    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:35.158060 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:35.158028    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:36.174830 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:36.174798    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:36.758023 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:36.757990    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:37.362389 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:37.362362    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:38.065066 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:38.065026    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:38.842628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:38.842590    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:39.446433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:39.446399    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:40.056315 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:40.056280    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:41.057106 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:41.057076    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:41.611218 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:41.611182    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:41.830707 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:41.830653    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:42.314912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:42.314882    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:42.480646 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:42.480611    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:42.663095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:42.663046    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:43.350652 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.350622    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:43.585832 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.585794    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"]
Oct 30 19:54:43.590430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.590402    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.599188 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.599150    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"]
Oct 30 19:54:43.599987 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.599962    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a89c71\"/\"simulator-single-main-a89c71-kserve-self-signed-certs\""
Oct 30 19:54:43.600115 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.600099    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a89c71\"/\"default-dockercfg-98ghw\""
Oct 30 19:54:43.600187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.600176    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a89c71\"/\"kube-root-ca.crt\""
Oct 30 19:54:43.600273 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.600257    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a89c71\"/\"openshift-service-ca.crt\""
Oct 30 19:54:43.685395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.685353    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tmp-dir\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.685395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.685390    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-dshm\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.685717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.685517    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjsr\" (UniqueName: \"kubernetes.io/projected/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-kube-api-access-ssjsr\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.685717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.685548    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-model-cache\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.685717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.685632    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tls-certs\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.685717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.685678    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-home\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.699664 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.699628    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:43.786960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.786923    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjsr\" (UniqueName: \"kubernetes.io/projected/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-kube-api-access-ssjsr\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.787179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.786978    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-model-cache\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.787179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.787030    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tls-certs\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.787179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.787080    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-home\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.787179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.787125    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tmp-dir\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.787179 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.787164    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-dshm\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.787640 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.787618    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-home\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.787771 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.787712    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tmp-dir\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.787901 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.787878    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-model-cache\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.790019 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.789988    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-dshm\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.790119 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.790059    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tls-certs\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.798676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.798653    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjsr\" (UniqueName: \"kubernetes.io/projected/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-kube-api-access-ssjsr\") pod \"simulator-single-main-a89c71-kserve-5bd758b779-k45nh\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") " pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:43.905994 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:43.905964    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:44.185048 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:44.185013    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"]
Oct 30 19:54:44.189718 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:54:44.189677    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb397898b_4bfe_4acd_9a84_c92ca9e7d77a.slice/crio-aecdaef6d8e0df70a70b8010a50b35d4f921ad88cf59272ceea451a60f45b9cf WatchSource:0}: Error finding container aecdaef6d8e0df70a70b8010a50b35d4f921ad88cf59272ceea451a60f45b9cf: Status 404 returned error can't find the container with id aecdaef6d8e0df70a70b8010a50b35d4f921ad88cf59272ceea451a60f45b9cf
Oct 30 19:54:44.268800 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:44.268748    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:44.726790 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:44.726753    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:54:44.775867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:44.775771    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh" event={"ID":"b397898b-4bfe-4acd-9a84-c92ca9e7d77a","Type":"ContainerStarted","Data":"c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0"}
Oct 30 19:54:44.775867 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:44.775826    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh" event={"ID":"b397898b-4bfe-4acd-9a84-c92ca9e7d77a","Type":"ContainerStarted","Data":"aecdaef6d8e0df70a70b8010a50b35d4f921ad88cf59272ceea451a60f45b9cf"}
Oct 30 19:54:44.795322 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:44.795280    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh" podStartSLOduration=1.795268106 podStartE2EDuration="1.795268106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:44.794826766 +0000 UTC m=+2919.697839805" watchObservedRunningTime="2026-10-30 19:54:44.795268106 +0000 UTC m=+2919.698281140"
Oct 30 19:54:44.906081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:44.906053    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:45.065903 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:45.065815    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:45.958000 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:45.957964    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:46.583137 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:46.583105    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:48.336951 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:48.336904    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:48.498203 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:48.498167    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:48.678382 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:48.678343    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:49.548207 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:49.548171    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:50.367909 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:50.367875    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:51.053483 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:51.053446    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:51.859505 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:51.859470    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:52.124296 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:54:52.124207    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:54:52.499194 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:52.499150    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:53.234187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:53.234153    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:53.872814 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:53.872779    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:53.906776 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:53.906744    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:53.919291 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:53.919267    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:54.490904 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:54.490871    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:54.835254 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:54.835175    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:54:55.296861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:55.296826    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:55.851442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:55.851410    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:56.551399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:56.551358    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:57.162390 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:57.162350    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:57.859080 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:57.859036    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:58.446758 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:58.446718    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:59.037702 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:59.037644    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:59.196662 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:59.196620    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:59.850433 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:54:59.850397    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:00.543439 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:00.543408    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:01.316431 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:01.316398    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:01.951990 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:01.951957    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:02.635235 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:02.635194    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:03.342252 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:03.342211    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:04.046565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:04.046528    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:04.667895 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:04.667856    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:05.294391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:05.294342    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:05.912223 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:05.912186    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:06.656586 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:06.656549    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:07.353955 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:07.353921    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:07.978633 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:07.978596    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:08.730715 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:08.730668    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:09.340980 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:09.340947    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:09.939942 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:09.939910    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:10.737798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:10.737748    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:11.416321 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:11.416281    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:12.022232 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:12.022196    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:12.054803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:12.054775    4925 scope.go:122] "RemoveContainer" containerID="7c0202327c8897b11c3531895b459198ec45da096aab80290e1ab70bd2ca5317"
Oct 30 19:55:12.063530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:12.063511    4925 scope.go:122] "RemoveContainer" containerID="8d6c26b9fd968c9cbc0440186b3236e702aee0cea8a07912ca3f61e5472cf148"
Oct 30 19:55:12.071564 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:12.071545    4925 scope.go:122] "RemoveContainer" containerID="4d3a787599574a3d5023c28655cf16edde8774546f64f7d0e0db25bf8b2d7e9e"
Oct 30 19:55:12.079724 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:12.079585    4925 scope.go:122] "RemoveContainer" containerID="db86b5c98db2b4c454aa8b0055d4db86023be3d18ae3c03a694cb244d1645c34"
Oct 30 19:55:12.087147 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:12.087128    4925 scope.go:122] "RemoveContainer" containerID="b5f56054f5cb9872faa95a3d1fcf3898f08439a8982ff9e027d9673b303d0457"
Oct 30 19:55:12.746278 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:12.746242    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:13.354550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:13.354511    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:13.958563 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:13.958520    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:14.769924 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:14.769891    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:14.997239 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:14.997207    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:15.861631 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:15.861597    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:16.642490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:16.642455    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:17.259577 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:17.259538    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:17.943749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:17.943715    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:18.660176 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:18.660143    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:19.267212 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:19.267175    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:19.920199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:19.920162    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:20.025313 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:20.025278    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"]
Oct 30 19:55:20.167303 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:20.167268    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:20.657340 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:20.657292    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:21.445305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:21.445275    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:22.063436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:22.063402    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:22.125380 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:55:22.125344    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:55:22.945856 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:22.945823    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:23.560042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:23.560003    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:24.174240 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:24.174198    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:24.940455 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:24.940425    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:25.172251 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:25.172220    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"]
Oct 30 19:55:25.363510 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:25.363428    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:25.530174 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:25.530140    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:25.740296 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:25.740258    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:26.380339 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:26.380304    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:30.173271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:30.173206    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz" podUID="bf998829-ac74-49b1-b3a0-be4b456030fd" containerName="payload-processing" containerID="cri-o://6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce" gracePeriod=25
Oct 30 19:55:32.726947 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:32.726900    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:32.906489 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:32.906446    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:33.111320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:33.111222    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:34.724835 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:34.724794    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz" podUID="bf998829-ac74-49b1-b3a0-be4b456030fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:55:35.046387 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.046291    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh" podUID="b397898b-4bfe-4acd-9a84-c92ca9e7d77a" containerName="main" containerID="cri-o://c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0" gracePeriod=45
Oct 30 19:55:35.294949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.294925    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:55:35.365539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.365475    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tmp-dir\") pod \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") "
Oct 30 19:55:35.365539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.365531    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-dshm\") pod \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") "
Oct 30 19:55:35.365798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.365592    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tls-certs\") pod \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") "
Oct 30 19:55:35.365798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.365625    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-kube-api-access-ssjsr\" (UniqueName: \"kubernetes.io/projected/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-kube-api-access-ssjsr\") pod \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") "
Oct 30 19:55:35.365798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.365667    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-model-cache\") pod \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") "
Oct 30 19:55:35.365798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.365713    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-home\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-home\") pod \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\" (UID: \"b397898b-4bfe-4acd-9a84-c92ca9e7d77a\") "
Oct 30 19:55:35.366033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.365828    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tmp-dir" pod "b397898b-4bfe-4acd-9a84-c92ca9e7d77a" (UID: "b397898b-4bfe-4acd-9a84-c92ca9e7d77a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:35.366033 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.365969    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-model-cache" pod "b397898b-4bfe-4acd-9a84-c92ca9e7d77a" (UID: "b397898b-4bfe-4acd-9a84-c92ca9e7d77a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:35.366156 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.366101    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-home" pod "b397898b-4bfe-4acd-9a84-c92ca9e7d77a" (UID: "b397898b-4bfe-4acd-9a84-c92ca9e7d77a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:35.366614 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.366587    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:35.366755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.366619    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:35.366755 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.366634    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:35.368187 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.368161    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-dshm" pod "b397898b-4bfe-4acd-9a84-c92ca9e7d77a" (UID: "b397898b-4bfe-4acd-9a84-c92ca9e7d77a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:35.368300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.368196    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tls-certs" pod "b397898b-4bfe-4acd-9a84-c92ca9e7d77a" (UID: "b397898b-4bfe-4acd-9a84-c92ca9e7d77a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:35.368300 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.368250    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-kube-api-access-ssjsr" pod "b397898b-4bfe-4acd-9a84-c92ca9e7d77a" (UID: "b397898b-4bfe-4acd-9a84-c92ca9e7d77a"). InnerVolumeSpecName "kube-api-access-ssjsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:35.467956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.467920    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:35.467956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.467950    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:35.468170 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:35.467966    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjsr\" (UniqueName: \"kubernetes.io/projected/b397898b-4bfe-4acd-9a84-c92ca9e7d77a-kube-api-access-ssjsr\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:36.023585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:36.023550    4925 generic.go:356] "Generic (PLEG): container finished" podID="b397898b-4bfe-4acd-9a84-c92ca9e7d77a" containerID="c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0" exitCode=0
Oct 30 19:55:36.024042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:36.023599    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"
Oct 30 19:55:36.024042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:36.023638    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh" event={"ID":"b397898b-4bfe-4acd-9a84-c92ca9e7d77a","Type":"ContainerDied","Data":"c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0"}
Oct 30 19:55:36.024042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:36.023711    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh" event={"ID":"b397898b-4bfe-4acd-9a84-c92ca9e7d77a","Type":"ContainerDied","Data":"aecdaef6d8e0df70a70b8010a50b35d4f921ad88cf59272ceea451a60f45b9cf"}
Oct 30 19:55:36.024042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:36.023733    4925 scope.go:122] "RemoveContainer" containerID="c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0"
Oct 30 19:55:36.032828 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:36.032811    4925 scope.go:122] "RemoveContainer" containerID="c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0"
Oct 30 19:55:36.033113 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:55:36.033093    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0\": container with ID starting with c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0 not found: ID does not exist" containerID="c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0"
Oct 30 19:55:36.033166 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:36.033121    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0"} err="failed to get container status \"c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0\": rpc error: code = NotFound desc = could not find container \"c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0\": container with ID starting with c08f33aaed95df1d6ac675abbd8e2e8dc88e5fea197e4912cfd594a2aa6326d0 not found: ID does not exist"
Oct 30 19:55:36.043894 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:36.043871    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"]
Oct 30 19:55:36.048298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:36.048280    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a89c71/simulator-single-main-a89c71-kserve-5bd758b779-k45nh"]
Oct 30 19:55:37.260679 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:37.260645    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b397898b-4bfe-4acd-9a84-c92ca9e7d77a" path="/var/lib/kubelet/pods/b397898b-4bfe-4acd-9a84-c92ca9e7d77a/volumes"
Oct 30 19:55:44.724388 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:44.724355    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz" podUID="bf998829-ac74-49b1-b3a0-be4b456030fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:55:47.278850 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:47.278817    4925 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-hcvdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:55:47.279271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:47.278879    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-hcvdj" podUID="68707443-c353-4c78-9d2f-fb9e915c6b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:55:50.429452 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:50.429385    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:55:50.506441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:50.506406    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf998829-ac74-49b1-b3a0-be4b456030fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf998829-ac74-49b1-b3a0-be4b456030fd-plugins-config-volume\") pod \"bf998829-ac74-49b1-b3a0-be4b456030fd\" (UID: \"bf998829-ac74-49b1-b3a0-be4b456030fd\") "
Oct 30 19:55:50.506634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:50.506533    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf998829-ac74-49b1-b3a0-be4b456030fd-kube-api-access-6gm6d\" (UniqueName: \"kubernetes.io/projected/bf998829-ac74-49b1-b3a0-be4b456030fd-kube-api-access-6gm6d\") pod \"bf998829-ac74-49b1-b3a0-be4b456030fd\" (UID: \"bf998829-ac74-49b1-b3a0-be4b456030fd\") "
Oct 30 19:55:50.506809 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:50.506787    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf998829-ac74-49b1-b3a0-be4b456030fd-plugins-config-volume" pod "bf998829-ac74-49b1-b3a0-be4b456030fd" (UID: "bf998829-ac74-49b1-b3a0-be4b456030fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:50.508804 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:50.508777    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf998829-ac74-49b1-b3a0-be4b456030fd-kube-api-access-6gm6d" pod "bf998829-ac74-49b1-b3a0-be4b456030fd" (UID: "bf998829-ac74-49b1-b3a0-be4b456030fd"). InnerVolumeSpecName "kube-api-access-6gm6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:50.608026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:50.607997    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf998829-ac74-49b1-b3a0-be4b456030fd-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:50.608026 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:50.608022    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gm6d\" (UniqueName: \"kubernetes.io/projected/bf998829-ac74-49b1-b3a0-be4b456030fd-kube-api-access-6gm6d\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:51.098338 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.098300    4925 generic.go:356] "Generic (PLEG): container finished" podID="bf998829-ac74-49b1-b3a0-be4b456030fd" containerID="6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce" exitCode=0
Oct 30 19:55:51.098546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.098346    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"
Oct 30 19:55:51.098546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.098409    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz" event={"ID":"bf998829-ac74-49b1-b3a0-be4b456030fd","Type":"ContainerDied","Data":"6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce"}
Oct 30 19:55:51.098546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.098457    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz" event={"ID":"bf998829-ac74-49b1-b3a0-be4b456030fd","Type":"ContainerDied","Data":"2b851f334ef8cdbb9ea0dc3232091a16316e0dda6b4883065fda3f31b0a9a4f7"}
Oct 30 19:55:51.098546 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.098486    4925 scope.go:122] "RemoveContainer" containerID="6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce"
Oct 30 19:55:51.107912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.107893    4925 scope.go:122] "RemoveContainer" containerID="6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce"
Oct 30 19:55:51.108207 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:55:51.108183    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce\": container with ID starting with 6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce not found: ID does not exist" containerID="6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce"
Oct 30 19:55:51.108308 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.108220    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce"} err="failed to get container status \"6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce\": rpc error: code = NotFound desc = could not find container \"6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce\": container with ID starting with 6f101147e68602c2456532a6029c32e5e18217c441660f1dc7e237bb1ee835ce not found: ID does not exist"
Oct 30 19:55:51.121787 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.121759    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"]
Oct 30 19:55:51.125572 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.125537    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a89c71-7456c5cfdcrrgz"]
Oct 30 19:55:51.260644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:51.260615    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf998829-ac74-49b1-b3a0-be4b456030fd" path="/var/lib/kubelet/pods/bf998829-ac74-49b1-b3a0-be4b456030fd/volumes"
Oct 30 19:55:52.124535 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:55:52.124498    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:55:56.913002 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:56.912963    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:58.237762 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:58.237714    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:59.091237 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:55:59.091186    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:04.333117 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:56:04.333077    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:06.455029 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:56:06.455002    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:56:06.481482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:56:06.481458    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 19:56:12.122680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:56:12.122644    4925 scope.go:122] "RemoveContainer" containerID="51c579f2ab27757514ec74ce5da5327cbb8dae2c149c3cb147b4fed4e0513257"
Oct 30 19:56:12.131698 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:56:12.131664    4925 scope.go:122] "RemoveContainer" containerID="6ad9bbc107c83781c1bf076fd56a1ad1b74cf232536485c1cdb70dec49dba919"
Oct 30 19:56:22.125189 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:56:22.125102    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:56:52.124278 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:56:52.124241    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:56:59.081983 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:56:59.081942    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:22.128679 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:57:22.128634    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:57:45.401571 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:45.401533    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:45.882523 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:45.882434    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:46.123949 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:46.123872    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:49.047654 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:49.047619    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:49.226063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:49.226027    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:50.523331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.523297    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:50.679996 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.679967    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"]
Oct 30 19:57:50.680329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.680317    4925 cpu_manager.go:414] "containerMap: removing container" podUID="b397898b-4bfe-4acd-9a84-c92ca9e7d77a" containerName="main"
Oct 30 19:57:50.680376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.680330    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b397898b-4bfe-4acd-9a84-c92ca9e7d77a" containerName="main"
Oct 30 19:57:50.680376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.680349    4925 cpu_manager.go:414] "containerMap: removing container" podUID="bf998829-ac74-49b1-b3a0-be4b456030fd" containerName="payload-processing"
Oct 30 19:57:50.680376 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.680354    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf998829-ac74-49b1-b3a0-be4b456030fd" containerName="payload-processing"
Oct 30 19:57:50.680478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.680425    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="b397898b-4bfe-4acd-9a84-c92ca9e7d77a" containerName="main"
Oct 30 19:57:50.680478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.680436    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf998829-ac74-49b1-b3a0-be4b456030fd" containerName="payload-processing"
Oct 30 19:57:50.683457 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.683441    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:57:50.694749 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.694723    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"]
Oct 30 19:57:50.704582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.704548    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2l99\" (UniqueName: \"kubernetes.io/projected/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-kube-api-access-t2l99\") pod \"payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp\" (UID: \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:57:50.704764 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.704599    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp\" (UID: \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:57:50.709257 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.709220    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-137da305-dockercfg-xjrlm\""
Oct 30 19:57:50.711472 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.711430    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-5310d3\""
Oct 30 19:57:50.805861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.805776    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2l99\" (UniqueName: \"kubernetes.io/projected/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-kube-api-access-t2l99\") pod \"payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp\" (UID: \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:57:50.805861 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.805810    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp\" (UID: \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:57:50.806453 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.806434    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp\" (UID: \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:57:50.817726 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.817682    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2l99\" (UniqueName: \"kubernetes.io/projected/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-kube-api-access-t2l99\") pod \"payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp\" (UID: \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:57:50.995562 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:50.995527    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:57:51.273526 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:51.273496    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"]
Oct 30 19:57:51.276804 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:57:51.276772    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4348e5_a868_4ace_bce3_0d30fe48f9a9.slice/crio-4da8244321f7b4ebb6dbce7f906d1ac8ec101bbbaa9dec71ddfce291de967800 WatchSource:0}: Error finding container 4da8244321f7b4ebb6dbce7f906d1ac8ec101bbbaa9dec71ddfce291de967800: Status 404 returned error can't find the container with id 4da8244321f7b4ebb6dbce7f906d1ac8ec101bbbaa9dec71ddfce291de967800
Oct 30 19:57:51.663697 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:51.663662    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp" event={"ID":"0a4348e5-a868-4ace-bce3-0d30fe48f9a9","Type":"ContainerStarted","Data":"773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942"}
Oct 30 19:57:51.664132 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:51.663732    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp" event={"ID":"0a4348e5-a868-4ace-bce3-0d30fe48f9a9","Type":"ContainerStarted","Data":"4da8244321f7b4ebb6dbce7f906d1ac8ec101bbbaa9dec71ddfce291de967800"}
Oct 30 19:57:51.682430 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:51.681423    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp" podStartSLOduration=1.681404796 podStartE2EDuration="1.681404796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:51.681104935 +0000 UTC m=+3106.584117972" watchObservedRunningTime="2026-10-30 19:57:51.681404796 +0000 UTC m=+3106.584417838"
Oct 30 19:57:51.759893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:51.759860    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:51.968644 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:51.968550    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:52.124271 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:57:52.124235    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:57:52.131343 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:52.131239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:52.667752 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:52.667718    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:57:52.986960 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:52.986837    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:53.711262 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:53.711200    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:54.449481 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:54.449452    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:55.088504 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:55.088466    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:55.781088 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:55.781044    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:56.951157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:56.951119    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:57.295681 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:57.295596    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:57.899250 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:57.899211    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:58.669241 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:58.669209    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:59.211284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:59.211249    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:59.447450 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:57:59.447415    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:00.059349 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:00.059315    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:00.831639 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:00.831602    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:01.358277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:01.358244    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:02.201127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:02.201092    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:02.868488 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:02.868458    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:03.446618 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:03.446571    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:03.675284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:03.675258    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:58:04.016418 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:04.016384    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:04.176768 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:04.176730    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:04.653746 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:04.653653    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:04.867838 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:04.867801    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:05.074890 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:05.074803    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:05.703449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:05.703410    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:06.409959 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.409929    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"]
Oct 30 19:58:06.413363 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.413346    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.418789 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.418754    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5310d3\"/\"kube-root-ca.crt\""
Oct 30 19:58:06.422871 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.422853    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5310d3\"/\"default-dockercfg-glpxv\""
Oct 30 19:58:06.423144 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.423126    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-5310d3\"/\"simulator-single-main-5310d3-kserve-self-signed-certs\""
Oct 30 19:58:06.428441 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.428417    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"]
Oct 30 19:58:06.476284 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.476243    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:06.490333 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.490306    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-5310d3\"/\"openshift-service-ca.crt\""
Oct 30 19:58:06.542114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.542086    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd54c\" (UniqueName: \"kubernetes.io/projected/13973a97-4a4f-418b-8fc6-f5fff82a17f0-kube-api-access-nd54c\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.542280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.542131    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-model-cache\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.542280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.542239    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-home\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.542280 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.542264    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tmp-dir\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.542399 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.542333    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-dshm\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.542449 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.542411    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tls-certs\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.643893 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.643855    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd54c\" (UniqueName: \"kubernetes.io/projected/13973a97-4a4f-418b-8fc6-f5fff82a17f0-kube-api-access-nd54c\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.644097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.643910    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-model-cache\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.644097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.643932    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-home\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.644097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.643948    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tmp-dir\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.644097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.643999    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-dshm\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.644097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.644019    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tls-certs\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.644443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.644363    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-home\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.644443 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.644433    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-model-cache\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.644553 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.644450    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tmp-dir\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.646527 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.646504    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-dshm\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.646708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.646674    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tls-certs\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.658931 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.658905    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd54c\" (UniqueName: \"kubernetes.io/projected/13973a97-4a4f-418b-8fc6-f5fff82a17f0-kube-api-access-nd54c\") pod \"simulator-single-main-5310d3-kserve-77d47655f8-pj7lp\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") " pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:06.708061 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.707993    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:06.724435 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:06.724404    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:07.009670 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:07.009591    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"]
Oct 30 19:58:07.013315 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:58:07.013284    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13973a97_4a4f_418b_8fc6_f5fff82a17f0.slice/crio-7d21b2eab7b4eca0bddb42296d23a76688eca9e2d5b51eb95013d9394ddff369 WatchSource:0}: Error finding container 7d21b2eab7b4eca0bddb42296d23a76688eca9e2d5b51eb95013d9394ddff369: Status 404 returned error can't find the container with id 7d21b2eab7b4eca0bddb42296d23a76688eca9e2d5b51eb95013d9394ddff369
Oct 30 19:58:07.171512 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:07.171483    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:07.740041 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:07.739995    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp" event={"ID":"13973a97-4a4f-418b-8fc6-f5fff82a17f0","Type":"ContainerStarted","Data":"ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284"}
Oct 30 19:58:07.740477 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:07.740048    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp" event={"ID":"13973a97-4a4f-418b-8fc6-f5fff82a17f0","Type":"ContainerStarted","Data":"7d21b2eab7b4eca0bddb42296d23a76688eca9e2d5b51eb95013d9394ddff369"}
Oct 30 19:58:07.759859 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:07.759807    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp" podStartSLOduration=1.759789766 podStartE2EDuration="1.759789766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:07.758642589 +0000 UTC m=+3122.661655628" watchObservedRunningTime="2026-10-30 19:58:07.759789766 +0000 UTC m=+3122.662802802"
Oct 30 19:58:07.970518 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:07.970479    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:08.724583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:08.724552    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:08.764436 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:08.764402    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:09.646932 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:09.646901    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:10.368003 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:10.367966    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:11.004621 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:11.004588    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:11.875607 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:11.875579    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:12.589199 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:12.589164    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:13.311267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:13.311235    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:13.981276 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:13.981239    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:14.594083 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:14.594052    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:15.363634 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:15.363599    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:15.979022 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:15.978985    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:16.716774 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:16.716735    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:16.725596 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:16.725569    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:16.738359 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:16.738339    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:16.794424 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:16.794401    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:16.956478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:16.956449    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:17.774541 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:17.774503    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:18.451319 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:18.451279    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:19.070123 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:19.070080    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:19.762277 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:19.762242    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:20.358331 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:20.358295    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:21.118467 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:21.118431    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:21.883583 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:21.883544    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:22.125616 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:58:22.125574    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:58:22.565797 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:22.565750    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:23.515329 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:23.515293    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:24.278030 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:24.277998    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:25.045676 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:25.045639    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:25.676462 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:25.676428    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:25.849813 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:25.849779    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:26.383798 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:26.383765    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:27.163953 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:27.163924    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:27.800434 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:27.800399    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:28.460562 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:28.460532    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:29.166485 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:29.166446    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:29.793127 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:29.793092    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:30.545972 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:30.545936    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:31.168975 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:31.168938    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:31.801923 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:31.801891    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:32.548663 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:32.548625    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:33.180138 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:33.180105    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:33.817740 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:33.817707    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:34.326337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:34.326302    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:34.532576 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:34.532541    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:35.267114 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:35.267079    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:36.060095 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:36.060056    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:36.724535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:36.724497    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:37.300015 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:37.299962    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:38.116717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:38.116655    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:38.829152 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:38.829116    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:39.220478 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:39.220448    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"]
Oct 30 19:58:39.363491 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:39.363458    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:39.675183 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:39.675150    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:40.300494 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:40.300458    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:40.961097 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:40.961064    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:41.590672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:41.590640    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:42.380985 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:42.380949    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:43.168298 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:43.168265    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:43.776732 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:43.776677    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:44.363701 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:44.363639    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:44.523063 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:44.523028    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"]
Oct 30 19:58:44.653672 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:44.653633    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:44.819482 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:44.819455    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:44.991246 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:44.991146    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:45.165470 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:45.165435    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:45.360852 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:45.360760    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:45.874603 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:45.874562    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:49.523320 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:49.523266    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp" podUID="0a4348e5-a868-4ace-bce3-0d30fe48f9a9" containerName="payload-processing" containerID="cri-o://773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942" gracePeriod=25
Oct 30 19:58:52.124897 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:58:52.124856    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:58:52.607089 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:52.607055    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:52.796228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:52.796196    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:53.037678 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:53.035181    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:53.673557 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:53.673521    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp" podUID="0a4348e5-a868-4ace-bce3-0d30fe48f9a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:58:54.240711 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.240651    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp" podUID="13973a97-4a4f-418b-8fc6-f5fff82a17f0" containerName="main" containerID="cri-o://ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284" gracePeriod=45
Oct 30 19:58:54.498605 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.498553    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:54.581548 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.581511    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-dshm\") pod \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") "
Oct 30 19:58:54.581765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.581555    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-home\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-home\") pod \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") "
Oct 30 19:58:54.581765 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.581699    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13973a97-4a4f-418b-8fc6-f5fff82a17f0-kube-api-access-nd54c\" (UniqueName: \"kubernetes.io/projected/13973a97-4a4f-418b-8fc6-f5fff82a17f0-kube-api-access-nd54c\") pod \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") "
Oct 30 19:58:54.581880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.581763    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tls-certs\") pod \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") "
Oct 30 19:58:54.581880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.581823    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-model-cache\") pod \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") "
Oct 30 19:58:54.581880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.581846    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-home" pod "13973a97-4a4f-418b-8fc6-f5fff82a17f0" (UID: "13973a97-4a4f-418b-8fc6-f5fff82a17f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:54.582042 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.582005    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tmp-dir\") pod \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\" (UID: \"13973a97-4a4f-418b-8fc6-f5fff82a17f0\") "
Oct 30 19:58:54.582165 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.582134    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-model-cache" pod "13973a97-4a4f-418b-8fc6-f5fff82a17f0" (UID: "13973a97-4a4f-418b-8fc6-f5fff82a17f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:54.582442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.582255    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tmp-dir" pod "13973a97-4a4f-418b-8fc6-f5fff82a17f0" (UID: "13973a97-4a4f-418b-8fc6-f5fff82a17f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:54.582442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.582384    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:54.582442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.582402    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:54.582442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.582410    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:54.584360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.584335    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13973a97-4a4f-418b-8fc6-f5fff82a17f0-kube-api-access-nd54c" pod "13973a97-4a4f-418b-8fc6-f5fff82a17f0" (UID: "13973a97-4a4f-418b-8fc6-f5fff82a17f0"). InnerVolumeSpecName "kube-api-access-nd54c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:54.584585 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.584555    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-dshm" pod "13973a97-4a4f-418b-8fc6-f5fff82a17f0" (UID: "13973a97-4a4f-418b-8fc6-f5fff82a17f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:54.584709 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.584598    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tls-certs" pod "13973a97-4a4f-418b-8fc6-f5fff82a17f0" (UID: "13973a97-4a4f-418b-8fc6-f5fff82a17f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:54.682847 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.682811    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13973a97-4a4f-418b-8fc6-f5fff82a17f0-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:54.683238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.682863    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd54c\" (UniqueName: \"kubernetes.io/projected/13973a97-4a4f-418b-8fc6-f5fff82a17f0-kube-api-access-nd54c\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:54.683238 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.682874    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13973a97-4a4f-418b-8fc6-f5fff82a17f0-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:54.961841 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.961806    4925 generic.go:356] "Generic (PLEG): container finished" podID="13973a97-4a4f-418b-8fc6-f5fff82a17f0" containerID="ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284" exitCode=0
Oct 30 19:58:54.962038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.961879    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"
Oct 30 19:58:54.962038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.961900    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp" event={"ID":"13973a97-4a4f-418b-8fc6-f5fff82a17f0","Type":"ContainerDied","Data":"ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284"}
Oct 30 19:58:54.962038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.961950    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp" event={"ID":"13973a97-4a4f-418b-8fc6-f5fff82a17f0","Type":"ContainerDied","Data":"7d21b2eab7b4eca0bddb42296d23a76688eca9e2d5b51eb95013d9394ddff369"}
Oct 30 19:58:54.962038 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.961967    4925 scope.go:122] "RemoveContainer" containerID="ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284"
Oct 30 19:58:54.970992 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.970974    4925 scope.go:122] "RemoveContainer" containerID="ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284"
Oct 30 19:58:54.971247 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:58:54.971226    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284\": container with ID starting with ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284 not found: ID does not exist" containerID="ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284"
Oct 30 19:58:54.971305 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.971256    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284"} err="failed to get container status \"ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284\": rpc error: code = NotFound desc = could not find container \"ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284\": container with ID starting with ad3e2333b357d871072761616a45219ddc54b20ab78c7b4141f47cbb1c026284 not found: ID does not exist"
Oct 30 19:58:54.983628 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.983603    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"]
Oct 30 19:58:54.986395 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:54.986374    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-5310d3/simulator-single-main-5310d3-kserve-77d47655f8-pj7lp"]
Oct 30 19:58:55.261247 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:55.261167    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13973a97-4a4f-418b-8fc6-f5fff82a17f0" path="/var/lib/kubelet/pods/13973a97-4a4f-418b-8fc6-f5fff82a17f0/volumes"
Oct 30 19:58:56.929337 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.929305    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"]
Oct 30 19:58:56.929748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.929726    4925 cpu_manager.go:414] "containerMap: removing container" podUID="13973a97-4a4f-418b-8fc6-f5fff82a17f0" containerName="main"
Oct 30 19:58:56.929748 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.929739    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13973a97-4a4f-418b-8fc6-f5fff82a17f0" containerName="main"
Oct 30 19:58:56.929881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.929816    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="13973a97-4a4f-418b-8fc6-f5fff82a17f0" containerName="main"
Oct 30 19:58:56.934265 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.934241    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:56.960967 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.960942    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fvzqm\""
Oct 30 19:58:56.974883 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.974851    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0ba273d9-kserve-self-signed-certs\""
Oct 30 19:58:56.983143 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.983125    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 19:58:56.983267 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.983150    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 19:58:56.992717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:56.992674    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"]
Oct 30 19:58:57.002565 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.002536    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/911eddbe-2590-4cf3-b87f-9b2b7960c858-tls-certs\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.002717 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.002634    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-home\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.002784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.002724    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-tmp-dir\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.002885 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.002868    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-model-cache\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.002937 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.002906    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-dshm\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.003024 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.003004    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6qtg\" (UniqueName: \"kubernetes.io/projected/911eddbe-2590-4cf3-b87f-9b2b7960c858-kube-api-access-q6qtg\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.101727 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.101615    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:57.103763 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.103608    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-model-cache\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.104037 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.104005    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-dshm\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.104191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.104076    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qtg\" (UniqueName: \"kubernetes.io/projected/911eddbe-2590-4cf3-b87f-9b2b7960c858-kube-api-access-q6qtg\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.104191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.104134    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/911eddbe-2590-4cf3-b87f-9b2b7960c858-tls-certs\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.104191 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.104164    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-home\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.104391 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.104198    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-tmp-dir\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.104673 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.104653    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-tmp-dir\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.104944 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.104925    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-model-cache\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.105598 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.105578    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-home\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.107361 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.107339    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-dshm\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.109031 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.109008    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/911eddbe-2590-4cf3-b87f-9b2b7960c858-tls-certs\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.121104 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.121081    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qtg\" (UniqueName: \"kubernetes.io/projected/911eddbe-2590-4cf3-b87f-9b2b7960c858-kube-api-access-q6qtg\") pod \"e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") " pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.246589 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.246506    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:57.525912 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.525837    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"]
Oct 30 19:58:57.528882 ip-10-0-130-164 kubenswrapper[4925]: W1030 19:58:57.528847    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod911eddbe_2590_4cf3_b87f_9b2b7960c858.slice/crio-83d0bb63ac5c9b9c4470a341cc3ff38fc0648b72bce7e854d36eea5d67a152d7 WatchSource:0}: Error finding container 83d0bb63ac5c9b9c4470a341cc3ff38fc0648b72bce7e854d36eea5d67a152d7: Status 404 returned error can't find the container with id 83d0bb63ac5c9b9c4470a341cc3ff38fc0648b72bce7e854d36eea5d67a152d7
Oct 30 19:58:57.986357 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.986320    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk" event={"ID":"911eddbe-2590-4cf3-b87f-9b2b7960c858","Type":"ContainerStarted","Data":"e11e209c56753341b0450d8e6914f53fc70ee87d44d7a36d1b5b4de348c5cccf"}
Oct 30 19:58:57.986784 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:57.986363    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk" event={"ID":"911eddbe-2590-4cf3-b87f-9b2b7960c858","Type":"ContainerStarted","Data":"83d0bb63ac5c9b9c4470a341cc3ff38fc0648b72bce7e854d36eea5d67a152d7"}
Oct 30 19:58:58.006530 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:58.006486    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk" podStartSLOduration=2.006469208 podStartE2EDuration="2.006469208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:58.005046663 +0000 UTC m=+3172.908059679" watchObservedRunningTime="2026-10-30 19:58:58.006469208 +0000 UTC m=+3172.909482248"
Oct 30 19:58:58.246956 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:58.246880    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:58:59.265426 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:58:59.265389    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:01.392780 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:01.392739    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:03.674081 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:03.674043    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp" podUID="0a4348e5-a868-4ace-bce3-0d30fe48f9a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:59:07.247150 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:07.247112    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:59:07.260539 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:07.260511    4925 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:59:08.060829 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:08.060793    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:59:09.776071 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:09.776046    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:59:09.847881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:09.847820    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-kube-api-access-t2l99\" (UniqueName: \"kubernetes.io/projected/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-kube-api-access-t2l99\") pod \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\" (UID: \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\") "
Oct 30 19:59:09.847881 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:09.847864    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-plugins-config-volume\") pod \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\" (UID: \"0a4348e5-a868-4ace-bce3-0d30fe48f9a9\") "
Oct 30 19:59:09.848271 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:09.848252    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-plugins-config-volume" pod "0a4348e5-a868-4ace-bce3-0d30fe48f9a9" (UID: "0a4348e5-a868-4ace-bce3-0d30fe48f9a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:09.850206 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:09.850187    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-kube-api-access-t2l99" pod "0a4348e5-a868-4ace-bce3-0d30fe48f9a9" (UID: "0a4348e5-a868-4ace-bce3-0d30fe48f9a9"). InnerVolumeSpecName "kube-api-access-t2l99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:09.948617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:09.948584    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2l99\" (UniqueName: \"kubernetes.io/projected/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-kube-api-access-t2l99\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.948617 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:09.948611    4925 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a4348e5-a868-4ace-bce3-0d30fe48f9a9-plugins-config-volume\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:10.059490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:10.059450    4925 generic.go:356] "Generic (PLEG): container finished" podID="0a4348e5-a868-4ace-bce3-0d30fe48f9a9" containerID="773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942" exitCode=0
Oct 30 19:59:10.059490 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:10.059490    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"
Oct 30 19:59:10.059770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:10.059527    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp" event={"ID":"0a4348e5-a868-4ace-bce3-0d30fe48f9a9","Type":"ContainerDied","Data":"773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942"}
Oct 30 19:59:10.059770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:10.059599    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp" event={"ID":"0a4348e5-a868-4ace-bce3-0d30fe48f9a9","Type":"ContainerDied","Data":"4da8244321f7b4ebb6dbce7f906d1ac8ec101bbbaa9dec71ddfce291de967800"}
Oct 30 19:59:10.059770 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:10.059620    4925 scope.go:122] "RemoveContainer" containerID="773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942"
Oct 30 19:59:10.070550 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:10.070532    4925 scope.go:122] "RemoveContainer" containerID="773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942"
Oct 30 19:59:10.072127 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:59:10.072100    4925 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942\": container with ID starting with 773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942 not found: ID does not exist" containerID="773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942"
Oct 30 19:59:10.072228 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:10.072132    4925 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942"} err="failed to get container status \"773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942\": rpc error: code = NotFound desc = could not find container \"773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942\": container with ID starting with 773d7d1697e956e8a7bac6c8ea57d003baa8ab1f6f8382df5787f71b683a9942 not found: ID does not exist"
Oct 30 19:59:10.081520 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:10.081494    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"]
Oct 30 19:59:10.085543 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:10.085524    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-5310d3-5cd75c848f8jtp"]
Oct 30 19:59:11.261358 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:11.261321    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4348e5-a868-4ace-bce3-0d30fe48f9a9" path="/var/lib/kubelet/pods/0a4348e5-a868-4ace-bce3-0d30fe48f9a9/volumes"
Oct 30 19:59:14.555442 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:14.555410    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:17.613772 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:17.606016    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:17.995803 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:17.995682    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:22.126456 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:59:22.126422    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:59:24.451839 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:24.451799    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:41.977360 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:41.977315    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"]
Oct 30 19:59:42.108157 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:42.108120    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:43.055582 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:43.055547    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:52.124819 ip-10-0-130-164 kubenswrapper[4925]: E1030 19:59:52.124782    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 19:59:57.008675 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.008521    4925 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk" podUID="911eddbe-2590-4cf3-b87f-9b2b7960c858" containerName="main" containerID="cri-o://e11e209c56753341b0450d8e6914f53fc70ee87d44d7a36d1b5b4de348c5cccf" gracePeriod=45
Oct 30 19:59:57.051149 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.051099    4925 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk" podUID="911eddbe-2590-4cf3-b87f-9b2b7960c858" containerName="main" probeResult="failure" output="Get \"https://10.132.1.56:8000/ready\": dial tcp 10.132.1.56:8000: connect: connection refused"
Oct 30 19:59:57.384493 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.384458    4925 generic.go:356] "Generic (PLEG): container finished" podID="911eddbe-2590-4cf3-b87f-9b2b7960c858" containerID="e11e209c56753341b0450d8e6914f53fc70ee87d44d7a36d1b5b4de348c5cccf" exitCode=0
Oct 30 19:59:57.384680 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.384581    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk" event={"ID":"911eddbe-2590-4cf3-b87f-9b2b7960c858","Type":"ContainerDied","Data":"e11e209c56753341b0450d8e6914f53fc70ee87d44d7a36d1b5b4de348c5cccf"}
Oct 30 19:59:57.505708 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.499843    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:59:57.592781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.592478    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/911eddbe-2590-4cf3-b87f-9b2b7960c858-tls-certs\" (UniqueName: \"kubernetes.io/secret/911eddbe-2590-4cf3-b87f-9b2b7960c858-tls-certs\") pod \"911eddbe-2590-4cf3-b87f-9b2b7960c858\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") "
Oct 30 19:59:57.592781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.592518    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911eddbe-2590-4cf3-b87f-9b2b7960c858-kube-api-access-q6qtg\" (UniqueName: \"kubernetes.io/projected/911eddbe-2590-4cf3-b87f-9b2b7960c858-kube-api-access-q6qtg\") pod \"911eddbe-2590-4cf3-b87f-9b2b7960c858\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") "
Oct 30 19:59:57.592781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.592603    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-dshm\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-dshm\") pod \"911eddbe-2590-4cf3-b87f-9b2b7960c858\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") "
Oct 30 19:59:57.592781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.592640    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-model-cache\") pod \"911eddbe-2590-4cf3-b87f-9b2b7960c858\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") "
Oct 30 19:59:57.592781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.592667    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-tmp-dir\") pod \"911eddbe-2590-4cf3-b87f-9b2b7960c858\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") "
Oct 30 19:59:57.592781 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.592774    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-home\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-home\") pod \"911eddbe-2590-4cf3-b87f-9b2b7960c858\" (UID: \"911eddbe-2590-4cf3-b87f-9b2b7960c858\") "
Oct 30 19:59:57.593224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.592944    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-tmp-dir" pod "911eddbe-2590-4cf3-b87f-9b2b7960c858" (UID: "911eddbe-2590-4cf3-b87f-9b2b7960c858"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:57.593224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.593116    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-home" pod "911eddbe-2590-4cf3-b87f-9b2b7960c858" (UID: "911eddbe-2590-4cf3-b87f-9b2b7960c858"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:57.593224 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.593148    4925 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-tmp-dir\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.593394 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.593305    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-model-cache" pod "911eddbe-2590-4cf3-b87f-9b2b7960c858" (UID: "911eddbe-2590-4cf3-b87f-9b2b7960c858"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:57.613846 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.613800    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/911eddbe-2590-4cf3-b87f-9b2b7960c858-tls-certs" pod "911eddbe-2590-4cf3-b87f-9b2b7960c858" (UID: "911eddbe-2590-4cf3-b87f-9b2b7960c858"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:57.625454 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.625412    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911eddbe-2590-4cf3-b87f-9b2b7960c858-kube-api-access-q6qtg" pod "911eddbe-2590-4cf3-b87f-9b2b7960c858" (UID: "911eddbe-2590-4cf3-b87f-9b2b7960c858"). InnerVolumeSpecName "kube-api-access-q6qtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:57.650192 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.650162    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-dshm" pod "911eddbe-2590-4cf3-b87f-9b2b7960c858" (UID: "911eddbe-2590-4cf3-b87f-9b2b7960c858"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:57.694535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.694507    4925 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-dshm\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.694535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.694532    4925 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-model-cache\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.694535 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.694542    4925 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/911eddbe-2590-4cf3-b87f-9b2b7960c858-home\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.694880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.694551    4925 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/911eddbe-2590-4cf3-b87f-9b2b7960c858-tls-certs\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.694880 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:57.694559    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6qtg\" (UniqueName: \"kubernetes.io/projected/911eddbe-2590-4cf3-b87f-9b2b7960c858-kube-api-access-q6qtg\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:58.393102 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:58.393078    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"
Oct 30 19:59:58.393606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:58.393075    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk" event={"ID":"911eddbe-2590-4cf3-b87f-9b2b7960c858","Type":"ContainerDied","Data":"83d0bb63ac5c9b9c4470a341cc3ff38fc0648b72bce7e854d36eea5d67a152d7"}
Oct 30 19:59:58.393606 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:58.393228    4925 scope.go:122] "RemoveContainer" containerID="e11e209c56753341b0450d8e6914f53fc70ee87d44d7a36d1b5b4de348c5cccf"
Oct 30 19:59:58.417255 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:58.417224    4925 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"]
Oct 30 19:59:58.422292 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:58.422268    4925 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0ba273d9-kserve-7748b75b89-xfnlk"]
Oct 30 19:59:58.871918 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:58.871826    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:59.043714 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:59.043237    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:59.277713 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:59.277561    4925 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911eddbe-2590-4cf3-b87f-9b2b7960c858" path="/var/lib/kubelet/pods/911eddbe-2590-4cf3-b87f-9b2b7960c858/volumes"
Oct 30 19:59:59.749716 ip-10-0-130-164 kubenswrapper[4925]: I1030 19:59:59.749317    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:06.167097 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.167066    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"]
Oct 30 20:00:06.167489 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.167435    4925 cpu_manager.go:414] "containerMap: removing container" podUID="911eddbe-2590-4cf3-b87f-9b2b7960c858" containerName="main"
Oct 30 20:00:06.167489 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.167445    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911eddbe-2590-4cf3-b87f-9b2b7960c858" containerName="main"
Oct 30 20:00:06.167489 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.167460    4925 cpu_manager.go:414] "containerMap: removing container" podUID="0a4348e5-a868-4ace-bce3-0d30fe48f9a9" containerName="payload-processing"
Oct 30 20:00:06.167489 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.167465    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4348e5-a868-4ace-bce3-0d30fe48f9a9" containerName="payload-processing"
Oct 30 20:00:06.167631 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.167531    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="911eddbe-2590-4cf3-b87f-9b2b7960c858" containerName="main"
Oct 30 20:00:06.167631 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.167540    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4348e5-a868-4ace-bce3-0d30fe48f9a9" containerName="payload-processing"
Oct 30 20:00:06.172097 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.172072    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.175457 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.175428    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"]
Oct 30 20:00:06.188773 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.188742    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:06.188773 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.188755    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:06.188990 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.188820    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:06.201087 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.201066    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z26xs\""
Oct 30 20:00:06.224464 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.224432    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae7039e4-dvgv6\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.224599 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.224469    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae7039e4-dvgv6\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.224643 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.224597    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t658m\" (UniqueName: \"kubernetes.io/projected/e32fbe47-291d-45aa-8aee-8107a043660d-kube-api-access-t658m\") pod \"maas-api-revoke-sub-ae7039e4-dvgv6\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.325183 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.325145    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t658m\" (UniqueName: \"kubernetes.io/projected/e32fbe47-291d-45aa-8aee-8107a043660d-kube-api-access-t658m\") pod \"maas-api-revoke-sub-ae7039e4-dvgv6\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.325348 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.325206    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae7039e4-dvgv6\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.325348 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.325228    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae7039e4-dvgv6\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.325912 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.325892    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae7039e4-dvgv6\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.327739 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.327723    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae7039e4-dvgv6\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.351545 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.351517    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t658m\" (UniqueName: \"kubernetes.io/projected/e32fbe47-291d-45aa-8aee-8107a043660d-kube-api-access-t658m\") pod \"maas-api-revoke-sub-ae7039e4-dvgv6\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.485800 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.485714    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:06.763653 ip-10-0-130-164 kubenswrapper[4925]: W1030 20:00:06.763623    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32fbe47_291d_45aa_8aee_8107a043660d.slice/crio-128221aebce1ecb667ebcbac65c6aa02609ef5019ccdddd3e7731a90f7d2e90a WatchSource:0}: Error finding container 128221aebce1ecb667ebcbac65c6aa02609ef5019ccdddd3e7731a90f7d2e90a: Status 404 returned error can't find the container with id 128221aebce1ecb667ebcbac65c6aa02609ef5019ccdddd3e7731a90f7d2e90a
Oct 30 20:00:06.767709 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:06.767673    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"]
Oct 30 20:00:07.465210 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:07.465176    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6" event={"ID":"e32fbe47-291d-45aa-8aee-8107a043660d","Type":"ContainerStarted","Data":"8a4cd643bdca95f97e2d93e4b0c559e099e541ff5f1fa08bdcc6f4248d37b3c2"}
Oct 30 20:00:07.465647 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:07.465230    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6" event={"ID":"e32fbe47-291d-45aa-8aee-8107a043660d","Type":"ContainerStarted","Data":"128221aebce1ecb667ebcbac65c6aa02609ef5019ccdddd3e7731a90f7d2e90a"}
Oct 30 20:00:07.526994 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:07.526948    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6" podStartSLOduration=1.52693566 podStartE2EDuration="1.52693566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:07.514812088 +0000 UTC m=+3242.417825125" watchObservedRunningTime="2026-10-30 20:00:07.52693566 +0000 UTC m=+3242.429948696"
Oct 30 20:00:08.471107 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:08.471070    4925 generic.go:356] "Generic (PLEG): container finished" podID="e32fbe47-291d-45aa-8aee-8107a043660d" containerID="8a4cd643bdca95f97e2d93e4b0c559e099e541ff5f1fa08bdcc6f4248d37b3c2" exitCode=0
Oct 30 20:00:08.471565 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:08.471111    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6" event={"ID":"e32fbe47-291d-45aa-8aee-8107a043660d","Type":"ContainerDied","Data":"8a4cd643bdca95f97e2d93e4b0c559e099e541ff5f1fa08bdcc6f4248d37b3c2"}
Oct 30 20:00:09.616007 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.615983    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:09.755445 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.755361    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32fbe47-291d-45aa-8aee-8107a043660d-kube-api-access-t658m\" (UniqueName: \"kubernetes.io/projected/e32fbe47-291d-45aa-8aee-8107a043660d-kube-api-access-t658m\") pod \"e32fbe47-291d-45aa-8aee-8107a043660d\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") "
Oct 30 20:00:09.755607 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.755488    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-openshift-ca\") pod \"e32fbe47-291d-45aa-8aee-8107a043660d\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") "
Oct 30 20:00:09.755607 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.755514    4925 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-xks-ca\") pod \"e32fbe47-291d-45aa-8aee-8107a043660d\" (UID: \"e32fbe47-291d-45aa-8aee-8107a043660d\") "
Oct 30 20:00:09.755871 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.755845    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-openshift-ca" pod "e32fbe47-291d-45aa-8aee-8107a043660d" (UID: "e32fbe47-291d-45aa-8aee-8107a043660d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:09.757727 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.757671    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-xks-ca" pod "e32fbe47-291d-45aa-8aee-8107a043660d" (UID: "e32fbe47-291d-45aa-8aee-8107a043660d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:09.757862 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.757727    4925 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32fbe47-291d-45aa-8aee-8107a043660d-kube-api-access-t658m" pod "e32fbe47-291d-45aa-8aee-8107a043660d" (UID: "e32fbe47-291d-45aa-8aee-8107a043660d"). InnerVolumeSpecName "kube-api-access-t658m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:09.856462 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.856424    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-openshift-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:09.856462 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.856457    4925 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32fbe47-291d-45aa-8aee-8107a043660d-maas-api-xks-ca\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:09.856462 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:09.856467    4925 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t658m\" (UniqueName: \"kubernetes.io/projected/e32fbe47-291d-45aa-8aee-8107a043660d-kube-api-access-t658m\") on node \"ip-10-0-130-164.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:10.480362 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:10.480326    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6" event={"ID":"e32fbe47-291d-45aa-8aee-8107a043660d","Type":"ContainerDied","Data":"128221aebce1ecb667ebcbac65c6aa02609ef5019ccdddd3e7731a90f7d2e90a"}
Oct 30 20:00:10.480362 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:10.480345    4925 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7039e4-dvgv6"
Oct 30 20:00:10.480650 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:10.480372    4925 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="128221aebce1ecb667ebcbac65c6aa02609ef5019ccdddd3e7731a90f7d2e90a"
Oct 30 20:00:12.532186 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:12.532093    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:12.703265 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:12.703225    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:22.124348 ip-10-0-130-164 kubenswrapper[4925]: E1030 20:00:22.124303    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 20:00:26.274191 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.274155    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7z827"]
Oct 30 20:00:26.274560 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.274543    4925 cpu_manager.go:414] "containerMap: removing container" podUID="e32fbe47-291d-45aa-8aee-8107a043660d" containerName="revoke-keys"
Oct 30 20:00:26.274560 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.274555    4925 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32fbe47-291d-45aa-8aee-8107a043660d" containerName="revoke-keys"
Oct 30 20:00:26.274647 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.274613    4925 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32fbe47-291d-45aa-8aee-8107a043660d" containerName="revoke-keys"
Oct 30 20:00:26.278457 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.278437    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.283107 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.283083    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 20:00:26.283107 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.283083    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f8msk\""
Oct 30 20:00:26.283284 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.283110    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:00:26.286274 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.286254    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:00:26.288299 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.288277    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7z827"]
Oct 30 20:00:26.383095 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.383061    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:00:26.405435 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.405401    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-maas-api-tls\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.405573 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.405442    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-metrics-certs\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.405573 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.405461    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfhl\" (UniqueName: \"kubernetes.io/projected/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-kube-api-access-jjfhl\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.405667 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.405589    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-openshift-service-ca\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.506295 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.506260    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-openshift-service-ca\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.506447 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.506328    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-maas-api-tls\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.506447 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.506358    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-metrics-certs\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.506447 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.506378    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfhl\" (UniqueName: \"kubernetes.io/projected/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-kube-api-access-jjfhl\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.506969 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.506938    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-openshift-service-ca\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.509053 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.509031    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-maas-api-tls\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.509141 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.509095    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-metrics-certs\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.518158 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.518136    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfhl\" (UniqueName: \"kubernetes.io/projected/b8d0b09b-a533-404d-95f1-c546d7fe3fd8-kube-api-access-jjfhl\") pod \"maas-api-595cc648c-7z827\" (UID: \"b8d0b09b-a533-404d-95f1-c546d7fe3fd8\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.592524 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.592460    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:26.869410 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.869264    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7z827"]
Oct 30 20:00:26.872384 ip-10-0-130-164 kubenswrapper[4925]: W1030 20:00:26.872354    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d0b09b_a533_404d_95f1_c546d7fe3fd8.slice/crio-c962ad86acbf2d983e2fd876d14ec25e4c9f3a53bd767afdccfa41fc9a71bc00 WatchSource:0}: Error finding container c962ad86acbf2d983e2fd876d14ec25e4c9f3a53bd767afdccfa41fc9a71bc00: Status 404 returned error can't find the container with id c962ad86acbf2d983e2fd876d14ec25e4c9f3a53bd767afdccfa41fc9a71bc00
Oct 30 20:00:26.874226 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:26.874196    4925 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:00:27.558851 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:27.558753    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827" event={"ID":"b8d0b09b-a533-404d-95f1-c546d7fe3fd8","Type":"ContainerStarted","Data":"f33c1a2b5582f84baef7aae6f9723643a9993229f23b142f094d4755a0575e92"}
Oct 30 20:00:27.559338 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:27.559027    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827" event={"ID":"b8d0b09b-a533-404d-95f1-c546d7fe3fd8","Type":"ContainerStarted","Data":"c962ad86acbf2d983e2fd876d14ec25e4c9f3a53bd767afdccfa41fc9a71bc00"}
Oct 30 20:00:27.559338 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:27.559082    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:27.579640 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:27.579569    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827" podStartSLOduration=1.169371705 podStartE2EDuration="1.579549375s" totalImagesPullingTime="410.17767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:27.577183519 +0000 UTC m=+3262.480196558" watchObservedRunningTime="2026-10-30 20:00:27.579549375 +0000 UTC m=+3262.482562414"
Oct 30 20:00:33.575856 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:33.575825    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-7z827"
Oct 30 20:00:52.124733 ip-10-0-130-164 kubenswrapper[4925]: E1030 20:00:52.124635    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 20:00:59.495522 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:00:59.495483    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:06.481931 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:06.481900    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 20:01:06.509669 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:06.509645    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 20:01:22.125401 ip-10-0-130-164 kubenswrapper[4925]: E1030 20:01:22.125363    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 20:01:24.664619 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:24.664588    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-jd9rn_b4bc7918-4f7f-4441-9498-b19b27eca300/manager/0.log"
Oct 30 20:01:24.877988 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:24.877950    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vzg9h_26772c06-318e-40d8-a330-fc84c521a37d/manager/0.log"
Oct 30 20:01:25.378281 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:25.378247    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jbk56_595b9742-b806-4d07-8a18-0197d1af422e/server/0.log"
Oct 30 20:01:26.304631 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:26.304598    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c94c66984-tv7w5_b74ba3b2-3630-4bd7-ab04-674a8795958e/manager/0.log"
Oct 30 20:01:26.325077 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:26.325053    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c94c66984-tv7w5_b74ba3b2-3630-4bd7-ab04-674a8795958e/copy-manifests/0.log"
Oct 30 20:01:27.710709 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:27.710660    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-7z827_b8d0b09b-a533-404d-95f1-c546d7fe3fd8/maas-api/0.log"
Oct 30 20:01:28.839290 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:28.839258    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ae7039e4-dvgv6_e32fbe47-291d-45aa-8aee-8107a043660d/revoke-keys/0.log"
Oct 30 20:01:31.328704 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:31.328658    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-qxmlg_27be6c75-203d-49a0-a643-0b2fa5951724/manager/0.log"
Oct 30 20:01:31.691454 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:31.691424    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-hzxfv_9a43fae4-0624-46f6-9ea8-39770dbf7afb/kuadrant-console-plugin/0.log"
Oct 30 20:01:33.103923 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:33.103886    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c5577pfxb_83805502-d8df-466d-a7c3-c842d05347d2/istio-proxy/0.log"
Oct 30 20:01:34.229086 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:34.229046    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cllrc_9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f/istio-proxy/1.log"
Oct 30 20:01:34.429358 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:34.429324    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-hcvdj_68707443-c353-4c78-9d2f-fb9e915c6b15/istio-proxy/0.log"
Oct 30 20:01:35.182126 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:35.182097    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vs4nx_d3021cf9-99e2-4612-82c5-3c9bc2a39e30/istio-proxy/0.log"
Oct 30 20:01:36.107873 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:36.107841    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddbc459d5-jmrh9_8c6e8d41-096c-4121-b9c9-a147a7ee010d/router/0.log"
Oct 30 20:01:36.555508 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:36.555472    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cllrc_9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f/istio-proxy/0.log"
Oct 30 20:01:39.235761 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:39.235727    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:48.788122 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:48.788082    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-jd9rn_b4bc7918-4f7f-4441-9498-b19b27eca300/manager/0.log"
Oct 30 20:01:48.997312 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:48.997284    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vzg9h_26772c06-318e-40d8-a330-fc84c521a37d/manager/0.log"
Oct 30 20:01:49.444560 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:49.444524    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jbk56_595b9742-b806-4d07-8a18-0197d1af422e/server/0.log"
Oct 30 20:01:50.345123 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:50.345096    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c94c66984-tv7w5_b74ba3b2-3630-4bd7-ab04-674a8795958e/copy-manifests/0.log"
Oct 30 20:01:50.366080 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:50.366054    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c94c66984-tv7w5_b74ba3b2-3630-4bd7-ab04-674a8795958e/manager/0.log"
Oct 30 20:01:50.478994 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:50.478963    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:51.537967 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:51.537929    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-7z827_b8d0b09b-a533-404d-95f1-c546d7fe3fd8/maas-api/0.log"
Oct 30 20:01:52.125245 ip-10-0-130-164 kubenswrapper[4925]: E1030 20:01:52.125208    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 20:01:52.650665 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:52.650634    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ae7039e4-dvgv6_e32fbe47-291d-45aa-8aee-8107a043660d/revoke-keys/0.log"
Oct 30 20:01:54.740645 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:54.740605    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-qxmlg_27be6c75-203d-49a0-a643-0b2fa5951724/manager/0.log"
Oct 30 20:01:55.121245 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:55.121168    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-hzxfv_9a43fae4-0624-46f6-9ea8-39770dbf7afb/kuadrant-console-plugin/0.log"
Oct 30 20:01:56.312544 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:56.312498    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c5577pfxb_83805502-d8df-466d-a7c3-c842d05347d2/istio-proxy/0.log"
Oct 30 20:01:57.440141 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:57.440111    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cllrc_9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f/istio-proxy/1.log"
Oct 30 20:01:57.643957 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:57.643927    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-hcvdj_68707443-c353-4c78-9d2f-fb9e915c6b15/istio-proxy/0.log"
Oct 30 20:01:58.400105 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:58.400064    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vs4nx_d3021cf9-99e2-4612-82c5-3c9bc2a39e30/istio-proxy/0.log"
Oct 30 20:01:59.304040 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:59.304012    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddbc459d5-jmrh9_8c6e8d41-096c-4121-b9c9-a147a7ee010d/router/0.log"
Oct 30 20:01:59.557873 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:01:59.557786    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:22.123881 ip-10-0-130-164 kubenswrapper[4925]: E1030 20:02:22.123794    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 20:02:52.125147 ip-10-0-130-164 kubenswrapper[4925]: E1030 20:02:52.125111    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 20:02:59.641776 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:02:59.641744    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:04.214226 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:04.214187    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:04.430871 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:04.430831    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:22.124074 ip-10-0-130-164 kubenswrapper[4925]: E1030 20:03:22.124037    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 20:03:25.335726 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:25.335671    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:25.496967 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:25.496909    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:28.926134 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:28.926103    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kj2xn_797c4f9e-bf72-41af-a1df-f39270f68e1a/global-pull-secret-syncer/0.log"
Oct 30 20:03:29.049432 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:29.049405    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2xnmc_6a4b6b51-385b-4e07-9eae-5d57cd734d39/connection-checker/0.log"
Oct 30 20:03:29.081596 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:29.081572    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-98bt4_3c9100be-82b2-466c-833c-3ed05703fba2/connection-checker/0.log"
Oct 30 20:03:29.113236 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:29.113210    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jkwnj_c91b7f8f-3427-4d04-a5bd-31792aed5075/connection-checker/0.log"
Oct 30 20:03:29.180926 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:29.180837    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log"
Oct 30 20:03:29.426903 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:29.426870    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-164.us-east-2.compute.internal_64f9ae4e88f69ebe611e7b19f38b4cc6/haproxy/0.log"
Oct 30 20:03:36.478372 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:36.478338    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-qxmlg_27be6c75-203d-49a0-a643-0b2fa5951724/manager/0.log"
Oct 30 20:03:36.559544 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:36.559511    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-hzxfv_9a43fae4-0624-46f6-9ea8-39770dbf7afb/kuadrant-console-plugin/0.log"
Oct 30 20:03:38.381974 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:38.381940    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-rmqbj_6996469c-158c-46af-9e06-a89a9f7137c0/cluster-monitoring-operator/0.log"
Oct 30 20:03:38.587744 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:38.587714    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-25vxx_f962e9e6-b6f6-40c5-a903-d44e902f40b9/node-exporter/0.log"
Oct 30 20:03:38.620932 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:38.620909    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-25vxx_f962e9e6-b6f6-40c5-a903-d44e902f40b9/kube-rbac-proxy/0.log"
Oct 30 20:03:38.650933 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:38.650915    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-25vxx_f962e9e6-b6f6-40c5-a903-d44e902f40b9/init-textfile/0.log"
Oct 30 20:03:39.381325 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.381294    4925 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"]
Oct 30 20:03:39.384828 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.384808    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.394114 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.394090    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmds\"/\"openshift-service-ca.crt\""
Oct 30 20:03:39.394237 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.394119    4925 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmds\"/\"kube-root-ca.crt\""
Oct 30 20:03:39.394237 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.394175    4925 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-trmds\"/\"default-dockercfg-mnw2j\""
Oct 30 20:03:39.395381 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.395361    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"]
Oct 30 20:03:39.505066 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.505038    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-podres\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.505271 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.505071    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4x5\" (UniqueName: \"kubernetes.io/projected/f00e504b-6ba0-426f-a63f-560b64d78d77-kube-api-access-zw4x5\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.505271 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.505121    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-sys\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.505271 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.505216    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-proc\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.505271 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.505260    4925 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-lib-modules\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.606553 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.606515    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-podres\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.606553 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.606555    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4x5\" (UniqueName: \"kubernetes.io/projected/f00e504b-6ba0-426f-a63f-560b64d78d77-kube-api-access-zw4x5\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.606873 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.606645    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-sys\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.606873 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.606681    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-proc\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.606873 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.606737    4925 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-lib-modules\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.606873 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.606787    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-sys\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.606873 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.606798    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-proc\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.606873 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.606799    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-lib-modules\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.606873 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.606737    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f00e504b-6ba0-426f-a63f-560b64d78d77-podres\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.619872 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.619842    4925 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4x5\" (UniqueName: \"kubernetes.io/projected/f00e504b-6ba0-426f-a63f-560b64d78d77-kube-api-access-zw4x5\") pod \"perf-node-gather-daemonset-kbzld\" (UID: \"f00e504b-6ba0-426f-a63f-560b64d78d77\") " pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.698736 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.698682    4925 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:39.973042 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:39.972971    4925 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"]
Oct 30 20:03:39.976595 ip-10-0-130-164 kubenswrapper[4925]: W1030 20:03:39.976570    4925 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf00e504b_6ba0_426f_a63f_560b64d78d77.slice/crio-f8962e71a7bb8d8b86e4d6ab9d38b293f1cbaa86585352ce35f22d8a9073243f WatchSource:0}: Error finding container f8962e71a7bb8d8b86e4d6ab9d38b293f1cbaa86585352ce35f22d8a9073243f: Status 404 returned error can't find the container with id f8962e71a7bb8d8b86e4d6ab9d38b293f1cbaa86585352ce35f22d8a9073243f
Oct 30 20:03:40.508818 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:40.508783    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld" event={"ID":"f00e504b-6ba0-426f-a63f-560b64d78d77","Type":"ContainerStarted","Data":"8d17fd86dda386a4ef99134852ee272fe8262d1f10c45b07e13c6c9596f6c96d"}
Oct 30 20:03:40.509263 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:40.508837    4925 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld" event={"ID":"f00e504b-6ba0-426f-a63f-560b64d78d77","Type":"ContainerStarted","Data":"f8962e71a7bb8d8b86e4d6ab9d38b293f1cbaa86585352ce35f22d8a9073243f"}
Oct 30 20:03:40.509263 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:40.508902    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:40.526817 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:40.526774    4925 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld" podStartSLOduration=1.5267620160000002 podStartE2EDuration="1.526762016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:40.525463156 +0000 UTC m=+3455.428476193" watchObservedRunningTime="2026-10-30 20:03:40.526762016 +0000 UTC m=+3455.429775072"
Oct 30 20:03:40.917597 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:40.917559    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-wqlmz_c9e20809-cbf6-491c-b6e9-371655cbd3c6/networking-console-plugin/0.log"
Oct 30 20:03:41.504187 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:41.504155    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/2.log"
Oct 30 20:03:41.512149 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:41.512124    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fskzc_664daa2d-471e-4380-bfdc-73df3a83ab3e/console-operator/3.log"
Oct 30 20:03:42.753547 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:42.753519    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fndkp_45274ad9-273b-467a-8bf1-27a876267294/volume-data-source-validator/0.log"
Oct 30 20:03:43.971201 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:43.971169    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dcpr_5250b4fe-4b08-4a7a-bfc7-d7343f202585/dns/0.log"
Oct 30 20:03:44.004551 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:44.004526    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dcpr_5250b4fe-4b08-4a7a-bfc7-d7343f202585/kube-rbac-proxy/0.log"
Oct 30 20:03:44.244480 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:44.244391    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g76xc_a50accc7-ce62-4f71-9a4a-aa128cfe10c4/dns-node-resolver/0.log"
Oct 30 20:03:44.994943 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:44.994913    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cnsll_9a69fe59-2039-43ff-a132-eeed44422412/node-ca/0.log"
Oct 30 20:03:46.048907 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:46.048820    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c5577pfxb_83805502-d8df-466d-a7c3-c842d05347d2/istio-proxy/0.log"
Oct 30 20:03:46.524271 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:46.524244    4925 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-trmds/perf-node-gather-daemonset-kbzld"
Oct 30 20:03:47.981416 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:47.981389    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cllrc_9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f/istio-proxy/0.log"
Oct 30 20:03:47.985062 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:47.985039    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cllrc_9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f/istio-proxy/1.log"
Oct 30 20:03:48.026302 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:48.026278    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-hcvdj_68707443-c353-4c78-9d2f-fb9e915c6b15/istio-proxy/0.log"
Oct 30 20:03:48.200243 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:48.200213    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vs4nx_d3021cf9-99e2-4612-82c5-3c9bc2a39e30/istio-proxy/0.log"
Oct 30 20:03:48.437765 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:48.437736    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddbc459d5-jmrh9_8c6e8d41-096c-4121-b9c9-a147a7ee010d/router/0.log"
Oct 30 20:03:49.962497 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:49.962468    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c5577pfxb_83805502-d8df-466d-a7c3-c842d05347d2/istio-proxy/0.log"
Oct 30 20:03:51.405392 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:51.405353    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cllrc_9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f/istio-proxy/0.log"
Oct 30 20:03:51.408917 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:51.408891    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-cllrc_9e1a9e9f-4030-4fc5-b7cc-a1be1aab6b4f/istio-proxy/1.log"
Oct 30 20:03:51.420431 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:51.420411    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nb5x6_a09e9550-86c1-474b-9774-50dd9189f3fa/serve-healthcheck-canary/0.log"
Oct 30 20:03:51.426361 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:51.426334    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-hcvdj_68707443-c353-4c78-9d2f-fb9e915c6b15/istio-proxy/0.log"
Oct 30 20:03:51.523246 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:51.523221    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-vs4nx_d3021cf9-99e2-4612-82c5-3c9bc2a39e30/istio-proxy/0.log"
Oct 30 20:03:51.663555 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:51.663482    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddbc459d5-jmrh9_8c6e8d41-096c-4121-b9c9-a147a7ee010d/router/0.log"
Oct 30 20:03:52.014987 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:52.014907    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-h6hw8_973eb9f4-c418-4ca6-9cb1-bd5b4980b83f/insights-operator/0.log"
Oct 30 20:03:52.017537 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:52.017512    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-h6hw8_973eb9f4-c418-4ca6-9cb1-bd5b4980b83f/insights-operator/1.log"
Oct 30 20:03:52.127592 ip-10-0-130-164 kubenswrapper[4925]: E1030 20:03:52.124678    4925 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x4lg8" containerName="extractor"
Oct 30 20:03:52.243655 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:52.243622    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4lg8_d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04/kube-rbac-proxy/0.log"
Oct 30 20:03:52.274450 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:52.274366    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4lg8_d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04/exporter/0.log"
Oct 30 20:03:52.310705 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:52.310659    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4lg8_d6a4e9c9-0315-4f33-8ca6-2f5ca5d86d04/extractor/0.log"
Oct 30 20:03:55.200547 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:55.200514    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-jd9rn_b4bc7918-4f7f-4441-9498-b19b27eca300/manager/0.log"
Oct 30 20:03:55.436990 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:55.436952    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c5577pfxb_83805502-d8df-466d-a7c3-c842d05347d2/istio-proxy/0.log"
Oct 30 20:03:56.175118 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:56.175083    4925 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24ddg_d11698df-8a69-42d9-96a9-ce0d5a37b003/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:56.632465 ip-10-0-130-164 kubenswrapper[4925]: I1030 20:03:56.632375    4925 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vzg9h_26772c06-318e-40d8-a330-fc84c521a37d/manager/0.log"